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Abstract 

Ecological  models  are  important  tools  for  planning  ecosystem  restoration 
and  management  activities.  Models  help  to  organize  thinking,  conceptualize 
understanding  of  complex  systems,  and  forecast  environmental  benefits 
that  may  result  from  proposed  restoration  and  management  actions.  This 
report  provides  information  to  guide  environmental  planers  in  selection, 
development,  evaluation,  and  documentation  of  ecological  models.  A 
number  of  critical  issues  are  addressed,  including  specifying  objectives  and 
formulating  a  sound  conceptual  model,  choosing  among  types  of  models, 
deciding  when  to  develop  a  new  model,  systematically  evaluating  the 
quantitative  model,  addressing  parameter  and  model  uncertainty, 
developing  sections  of  the  model  through  iteration,  analyzing  alternatives, 
and  documenting  results.  Quantitative  modeling  is  shown  to  be  a  dynamic 
process  that  is  best  served  using  an  iterative  approach.  In  practice, 
individual  parts  of  a  conceptual  model  are  quantified  and  evaluated  in  a 
stepwise  fashion  until  the  entire  model  is  captured  quantitatively.  This 
iterative  approach  creates  transparency  in  model  development,  which  can 
remove  the  “black-box”  stigma  that  has  been  associated  with  the  use  of 
models  in  the  environmental  sciences. 
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1  Introduction 

Background 

Over  the  last  several  decades,  models  have  become  an  important  tool  in 
environmental  decision  making;  most  modern  environmental  projects 
require  models.  Although  quantitative  models  are  often  required  for 
projects,  there  is  still  considerable  apprehension  when  applying  models  to 
environmental  problems.  Ecosystems  are  inherently  complex,  and  for  any 
given  environmental  system,  the  number  of  interacting  factors  is  large 
(e.g.,  weather,  species,  hydrology,  geomorphology,  anthropogenic  factors, 
etc.).  Each  factor  operates  at  different  spatial  and  temporal  scales. 
Therefore,  it  is  rare  to  have  a  complete  dataset  that  encompasses  all  the 
different  permutations  of  environmental  conditions,  yet  planning  activities 
must  often  determine  how  a  system  will  respond  under  extreme  conditions 
-  i.e.,  outside  the  range  of  available  field  data.  Modeling  is  an  excellent 
tool  for  analyzing  environmental  systems  because  it  helps  to  fill  data  gaps 
and  provides  a  mechanism  to  systematically  compare  scenarios  across  a 
broad  range  of  conditions  that  would  not  be  possible  in  the  field. 

The  process  of  modeling  helps  researchers  to  organize  their  thoughts  and 
research  direction,  facilitates  communication,  and  creates  a  level  of  trans¬ 
parency  that  can  dispel  the  myth  that  models  are  “black-box”  endeavors. 
However,  models  do  have  limitations  and  should  not  be  viewed  as  all- 
inclusive  or  as  a  panacea  for  environmental  decision  making.  Since  environ¬ 
mental  models  are  simplified  representations  of  complex  systems,  they  are 
often  built  using  assumptions  regarding  the  unknown  components  in  the 
model.  The  usefulness  of  a  model  hinges  on  understanding  whether  the  data 
and  assumptions  used  to  develop  the  model  are  sufficient  enough  to  inform 
decisions.  It  is  also  important  to  understand  that  models  should  inform,  not 
dictate,  decision-making  processes  (Glaser  and  Bridges  2007).  Modeling 
does  not  provide  all  the  answers,  rather  good  models  should  inform 
management/planning  decisions  and  their  results  should  be  incorporated 
with  results  from  field  studies  and  professional  judgment  (by  subject  matter 
experts)  before  final  decisions  are  made. 
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Model  types 

Several  types  of  models  have  been  used  for  determining  environmental 
benefits,  ranging  from  simple  empirical  relations  describing  the  expected 
habitat  preferences  of  species  to  complex,  dynamic  models  of  material  flow 
(e.g.,  water,  sediment,  etc.)  to  agent-based  or  spatially  explicit  models  that 
address  large-scale  dynamics  (e.g.,  Foran  et  al.  2011;  Guisan  and 
Zimmermann  2000).  Some  projects  use  multiple  types  of  models  to  address 
their  questions.  In  general,  the  models  used  to  assess  environmental 
benefits  fall  into  six  basic  classes  (analytical,  conceptual,  index-based, 
simulation,  statistical,  and  spatial),  each  with  their  strengths  and 
weaknesses.  There  is  significant  literature  available  discussing  different 
types  of  models,  so  only  a  brief  description  of  each  model  type  will  be  given 
(for  more  detail  refer  to  Wissel  1992,  Grimm  1994,  Grant  1998,  Ford  1999, 
Peck  2000,  Jorgensen  and  Bendoricchio  2001,  Grimm  and  Railsback  2005, 
Fischenich  2008,  Grant  and  Swannack  2008,  Jorgensen  and  Fath  2011). 
Table  1  provides  a  brief  summary  of  model  types,  general  usage,  and 
examples.  Regardless  of  the  type  of  model,  all  models  should  be  developed 
using  the  methodology  explained  below. 


Table  1.  Description  of  model  types  often  used  for  modeling  environmental  benefits. 


Model 

General  Use 

Example 

Analytical 

Systems  where  solution  to  closed 
form  equations  represent  system 

Population  growth,  Lotka- 
Volterra  models 

Conceptual 

Diagramming  relationships  among 
components,  organizing  information, 
determining  data  needs 

CEMCAT  (see  Fischenich 
2008,  for  more  examples) 

Index 

Determining  habitat  quality  across  a 
landscape,  relates  species  presence 
to  environmental  variables 

HSI,  HGM 

Simulation 

Modeling  dynamics  of  complex 
systems  that  have  multiple  factors 
interacting  across  scales,  often  have 
spatial  components 

Agent-based  models,  ADH- 
CASM,  ELAM,  ICM,  system 
dynamic  models 

Statistical 

Analysis  of  datasets  to  determine 
distributional  properties  of  the  data 

AN  OVA,  goodness-of-fit, 
regression,  t-test, 

Spatial 

Projects  where  particular  spatial 
attributes  are  important  can  be 
incorporated  into  simulation  models 

GIS,  EDYS 

Analytical  models  are  models  for  which  a  specific  mathematical  form 
can  be  written  as  an  equation  or  set  of  equations.  These  models  are 
solvable  in  “closed  form”  -  they  have  a  general  solution  that  applies  to 
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situations  that  the  model  can  represent.  Common  types  of  analytical 
models  are  represented  by  differential  or  difference  equations  (or  a  series 
of  those  equations). 1  An  example  of  an  analytical  model  is  the  equation  for 
exponential  population  growth 


N  =  er[t~c)  (1) 

Using  this  model,  the  population  size  (JV)  can  be  calculated  for  any  time- 
step  (f),  just  knowing  the  population  size  and  the  rate  of  growth  (r),  where 
c  represents  a  constant  and  e  is  the  base  of  the  natural  logarithm. 

Conceptual  models  represent  the  system  of  interest  qualitatively,  usually 
as  a  diagram  showing  the  relationships  among  important  variables.  A  wide 
variety  of  approaches  are  used  for  developing  conceptual  models,  ranging 
from  simple  depictions  that  show  the  system’s  components  or  connections, 
to  others  that  imply  the  first  steps  towards  quantitative  model  development 
(Jorgensen  and  Fath  2011).  It  is  almost  impossible  to  quantitatively  model 
an  environmental  system  without  a  conceptual  model.  Conceptual  models 
can  serve  as  templates  for  quantitative  models,  allowing  researchers  to 
visualize  their  system  and  identify  important  flow  paths  and  feedback  loops. 
However,  conceptual  models  cannot  be  used  to  forecast  system  dynamics  or 
to  quantitatively  compare  scenarios.  There  has  been  considerable  work 
describing  the  different  approaches  used  for  developing  conceptual  models 
(Ford  1999,  Peck  2000,  Fischenich  2008,  Grant  and  Swannack  2008, 
Jorgensen  and  Fath  2011). 

Index  models  are  commonly  used  for  planning  and  restoration  studies. 
Briefly,  an  index  model  is  intended  to  translate  quantitatively  based 
measures  of  individual  habitat  features  or  processes  into  a  relative 
assessment  of  habitat  suitability  or  ecosystem  condition  (Tirpak  et  al. 

2009).  Two  common  index-based  approaches  are  habitat  suitability  index 
(HSI)  models  and  the  hydrogeomorphic  (HGM)  approach  that  is  used  to 
rapidly  assess  wetland  function.  The  HGM  methodology  is  thoroughly 


1  Differential  and  difference  equations  are  different  mathematical  expressions  of  the  same  model.  A 
differential  equation  expresses  the  relationship  between  a  dependent  variable  (y)  and  one  or  more 
independent  variables  (x's)  in  terms  of  rates  of  change  (r)  -  e.g.,  population  size  (N)  at  time  “t”  is  a 
function  of  the  change  in  N  with  respect  to  time  (r=dN/dt).  A  difference  equation  uses  a  sequence  of 
numerical  differences  to  compute  an  output  (y)  at  time  “t”  based  on  past  and  present  input  of  the 
independent  variable  (x’s)  and  past  outputs  of  the  dependent  variable  (y)  within  the  time  frame  of 
interest  --  e.g.,  N  at  time  "t”  is  a  function  of  N  at  (t-1)  plus  the  portion  of  new  N  added  during  that 
interval. 
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documented  and  has  been  broadly  applied  throughout  USACE  districts 
(Brinson  1993,  Smith  et  al.  1995,  Smith  2001)  and  will  not  be  discussed  in 
detail.1  The  HSI  approach  quantitatively  relates  potential  for  species 
presence  to  habitat  characteristics.  Species  have  complex  relationships  with 
their  environment  and  HSI  models  provide  a  simple  method  for  charac¬ 
terizing  potential  for  habitat  to  support  target  species/communities  across  a 
landscape.  In  general,  species-habitat  associations  are  scaled  from  o  to  1, 
with  1  representing  a  favorable  or  “ideal”  relationship  that  represents 
habitat  quality  at  a  particular  location.  Multiple  indices  can  be  combined 
into  a  composite  score  or  single  indices  can  be  used  to  interpret  habitat 
quality.  For  example,  an  oyster  HSI  relates  potential  for  oyster  presence  to 
bottom  substrate  and  mean  salinity,  among  other  factors  (Cake  1983)  to 
determine  habitat  suitability  in  the  Gulf  of  Mexico.  The  quantitative 
relationships  between  species  and  habitat  that  are  used  to  develop  these 
models  are  generally  based  on  literature,  field  studies,  or  expert  opinion. 
Understanding  the  relationships  between  species  and  habitat  is  a  complex 
and  evolving  field  of  study.  Likewise,  HSI  models  should  be  viewed  as 
adaptable  hypotheses  of  species-habitat  relationships  rather  than  actual 
cause-effect  relationships.2  Index-based  models  are  valuable  because  they 
serve  as  a  foundation  for  improved  decision-making  and  increased  under¬ 
standing  of  habitat  relationships  because  the  species-habitat  relationships 
represent  specific  hypotheses  that  can  be  tested  and  improved. 

Simulation,  or  process-based,  models  are  typically  computer 
programs  designed  to  numerically  represent  the  behavior  of  a  system  and 
its  characteristic  component  elements  and  processes  (in  environmental 
simulations,  these  models  describe  how  environmental  conditions  change 
across  time  and  may  involve  integrating  multiple  disciplines  (e.g.,  hydro¬ 
dynamics,  population  dynamics,  climate,  etc).  These  models  are  developed 
to  capture  and  mimic  real-world  systems,  that  unlike  real  systems,  can  be 
experimented  upon  at  various  scales  with  minimal  risk  to  the  real  environ¬ 
ment  (Peck  2004).  The  opportunity  presented  by  models  is  capacity  to  more 
rapidly  test  a  broader  array  of  alternatives  without  risk  to  the  environment 
(or  lost  investment).  Simulation  models  represent  the  relevant  aspects  of  a 
system  -  e.g.,  drivers,  stressors,  and  ecological  outputs  and  attributes  that 
may  be  subject  to  management  influence.  Simulation  models  are  often 
complex,  consisting  of  many  interacting  variables,  but  one  of  their  strengths 
is  that  they  provide  a  tool  with  which  to  understand  complex  systems. 


1  http://el.erdc.usace.armv.mil/wetlands/ guidebooks.cfm 

"  Note  that  a  perfect  quality  index  score  may  not  translate  to  increased  species  presence/utilization. 
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Examples  of  simulation  models  include  agent-based  models  (Grimm  and 
Railsback  2005,  Goodwin  et  al.  2006),  system  dynamics  models  (Odum 
1983,  Ford  1999,  Grant  and  Swannack  2008)  and  coupled  hydrodynamic- 
ecological  models  (Cerco  and  Noel  2010,  Dalyander  and  Cerco  2010). 

Spatial  models  are  any  models  that  incorporate  a  spatial  dimension, 
which  can  be  represented  implicitly  or  explicitly.  In  spatially  implicit 
models,  the  specific  locations  are  not  defined  and  are  represented  as 
number  or  proportion  of  sites  within  a  given  attribute.  Spatially  explicit 
models,  where  specific,  geo-referenced  locations  are  considered,  are  more 
common  in  environmental  benefits  analysis.  These  models  generally 
include  use  of  a  geographic  information  system  (GIS)  and  can  be  index- 
based  (e.g.,  if  an  HSI  model  is  designed  to  output  habitat  suitability  maps) 
or  can  be  included  as  part  of  a  simulation  model.  Environmental  models 
often  have  a  spatial  component  associated  with  them  (i.e.,  focusing  on  a 
specific  place)  and  spatial  models  are  often  treated  as  components  of 
larger  models. 

Statistical  models  are  data-driven  and  represent  the  distributional 
properties  of  the  data.  In  general,  these  types  of  models  identity  how  well 
data  can  be  explained  by  a  set  of  factors  or  how  well  variables  correlate  to 
each  other.  These  models  do  not  explicitly  address  causation;  rather,  they 
provide  tools  that  explore  how  well  a  given  set  of  data  can  be  explained  by 
a  suite  of  factors  (e.g.,  the  relationship  between  water  quality  or  bird 
species  diversity  and  the  width  and  vegetation  density  of  riparian  buffer 
strips  along  a  stream  reach).  Common  types  of  statistical  models  include 
linear  or  nonlinear  regression,  analysis  of  variance  (ANOVA),  analysis  of 
covariance  (ANCOVA),  contingency  tables,  t-tests,  and  goodness-of-fit 
tests,  among  others. 
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2  Model  Selection 

Two  major  issues  must  be  considered  in  selecting  an  appropriate  model. 
First,  the  project  team  should  align  model  selection  criteria/process  with 
the  problems  and  opportunities  to  be  addressed  and  associated  information 
demands  of  the  study/project.1  The  team  should  thoroughly  discuss  project 
scope,  scale  and  objectives,  ecosystem  characteristics,  availability  of  data, 
and  project  duration  with  subject  matter  experts,  stakeholders,  and 
staff/leadership  of  involved  agencies.  During  these  discussions,  it  is 
common  for  people  to  have  or  form  expectations  that  may  or  may  not  be 
realistic,  seek  complexity  that  may  or  may  not  be  necessary,  and  be  familiar 
with  a  particular  type  or  “brand”  of  model  and  request  that  it  be  used. 
However,  it  is  important  to  recognize  that  many  different  types  of  models 
can  be  used  to  address  environmental  problems  and  familiarity  with  a 
particular  type  of  model  does  not  indicate  its  usefulness  for  a  specfic  project. 
The  authors  recommend  adhering  to  the  principle  of  parsimony  when 
choosing  a  type  of  model,  that  is,  given  a  choice  between  multiple, 
appropriate  model  types,  choose  the  simplest  approach  that  can  address  the 
problem  sufficiently.  For  example,  when  trying  to  determine  where  to  best 
place  oyster  beds  for  restoration,  a  large-scale,  ecosystem-hydraulic 
modeling  approach  might  not  provide  better  answers  compared  to  a  much 
simpler  HSI  type  model,  which  requires  less  resources  (i.e.,  time,  effort  and 
cost)  compared  to  a  more  complex  model.  Even  when  problems,  oppor¬ 
tunities,  and  expectations  are  well-defined  prior  to  model  selection  (and 
frequently  they  are  not),  choosing  the  right  model  for  a  given  problem  is  not 
an  easy  task.  Foran  et  al.  (2011)  identified  some  of  the  issues  involved  with 
selecting  models  for  ecological  forecasting  and  used  the  relationships 
among  prescriptive  utility,  model  type,  and  effort  involved  in  model 
development  to  help  narrow  down  the  choices  of  models  for  a  given  project 
(see  Figure  2  in  Foran  et  al.  (2011)). 

The  second  issue  that  project  delivery  teams  will  face  is  whether  or  not  to 
develop  a  new  model  for  each  project  or  to  use  an  existing  model.  Speci¬ 
fically,  the  issue  is  the  degree  to  which  existing  models  of  the  system  or  of 
similar  systems  can  be  applied  to  the  project  to  yield  applicable  information 
of  adequate  quality.  In  an  extreme  case,  a  model  may  not  exist  that  can  be 


1  Step  1  of  the  Corps  planning  process;  see  Engineer  Regulation  1105-2-100,  Planning  Guidance 
Notebook,  22  April  2000,  http://www.iwr.usace.army.mil/docs/11052100.pdf 
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adapted,  and  a  completely  new  model  is  necessary.  Developing  a  new  model 
maximizes  the  likelihood  that  the  model  will  be  capable  of  addressing  the 
questions  for  that  specific  project.  However,  model  development  requires 
labor,  funding,  and  effort  that  might  exceed  the  resources  or  life  cycle  of 
many  projects.  Using  an  existing  model  might  decrease  the  time  it  takes  to 
prepare  a  model  for  application,  but  given  the  uniqueness  of  each  environ¬ 
mental  issue,  existing  models  may  need  to  be  adapted  (in  some  instances, 
significantly)  to  be  appropriate  for  application  to  different  project  settings. 

A  model’s  existing  certification  or  approval  of  a  model  for  use1  is  no 
guarantee  that  it  is  (or  will  be)  appropriate  for  application  to  all  study/ 
project  settings.  It  is  paramount  to  thoroughly  understand  the  intended 
uses,  assumptions,  limitations,  and  data  requirements  for  any  model, 
certified  or  not,  during  its  consideration  for  application  to  a  project’s 
specific  setting  and  context.  It  might  also  be  worthwhile  to  consider  regional 
demands  for  a  planning  model  that  has  broad  applicability  to  comparable 
ecosystems  throughout  a  given  ecological  region  (e.g.,  McKay  and  Pruitt,  in 
preparation).  Such  demands  might  suggest  there  are  benefits  to  be  gained 
via  development  of  a  regional  model,  where  costs  of  development  might  be 
distributed  among  its  potential  beneficiaries.  The  Ecosystem  Restoration 
Gateway  maintains  an  online  library  of  planning  models  that  includes  basic 
information  about  each  including  their  certification  status  (http://cw- 
environment.usace.army.mil/model-librarv.cfm?CoP=Restore&Option=Start).  Users  can  refer  to 
that  link  or  contact  the  National  Ecosystem  Planning  Center  of  Expertise 
(ECO-PCX,  http://el.erdc.usace.armv.mil/ecocx/index.cfm)  to  retrieve  or  offer 
information  about  cataloged  models  or  models  that  could  be  added  to  the 
Ecosystem  Restoration  Gateway  library. 


1  Engineer  Circular  1105-2-412,  “Assuring  the  Quality  of  Planning  Models”  (USACE  2011),  reflects  USACE 
policy  requiring  use  of  certified  or  approved  models  for  all  planning  activities.  It  is  important  to  note 
that  this  requirement  is  not  to  be  interpreted  as  universally  excluding  from  consideration  models  that 
have  not  yet  been  reviewed  and  certified/approved  for  use.  Rather  the  policy  reflects  a  requirement 
that  models  used  in  support  of  planning  decisions  are  to  be  reviewed  for  their  capacity  to  yield  reliable 
information  within  a  specified  context/setting  prior  to  arrival  at  a  planning  decision. 
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3  Model  Development 

This  chapter  focuses  on  guiding  users  through  phases  of  model 
development.  The  steps  described  below  generally  follow  Corps 
guidelines;1  however,  the  terminology  used  here  follows  that  generally 
accepted  in  the  field  of  ecological  modeling.  Once  a  specific  problem  has 
been  identified  and  both  the  planning  and  modeling  objectives  have  been 
clearly  defined,  the  basic  approach  is  as  follows: 

•  Develop  a  conceptual  model  identifying  the  specific  cause-effect 
relationships  among  important  components  of  the  system  of  interest. 

•  Quantify  these  relationships  based  on  analysis  of  the  best  information 
possible,  which  can  include  scientific  data  or  expert  opinion. 

•  Evaluate  the  information  yielded  by  the  model  in  terms  of  its  ability  to 
yield  information  that  describes  or  emulates  system  behavior. 

•  Apply  the  model  to  address  questions  regarding  the  effects  of  particular 
project  alternatives. 

•  Perform  periodic  post-audits  of  model  applications  to  manage 
confidence  in  the  model  and  the  information  it  yields  (this  will  be 
incredibly  important  as  adaptive  management  practices  are 
implemented). 

In  practice,  model  development  does  not  proceed  continuously  from  the 
conceptual  model  to  model  application.  Rather,  as  described  in  Chapter  7, 
Modeling  in  Practice,”  model  development  iterates  through  a  series  of 
intermediate  developmental  phases  (each  a  more  mature  form  of  its 
predecessor,  and  sometimes  halting  further  development  because  informa¬ 
tion  needs  are  found  to  have  been  met).  However,  in  general,  a  model  is  first 
conceptualized,  then  quantified  and  evaluated  as  the  model  becomes  fully 
integrated  into  a  form  that  can  be  applied  for  its  intended  use  (Figure  1). 
Model  documentation  should  be  developed  and  modified  throughout  each 
stage  of  a  model’s  development,  which  not  only  facilitates  compiling 
documentation  at  the  end  of  the  development  activity,  but  also  forces  the 
model  development  team  to  more  thoroughly  understand  relationships 
among  and  between  model  development  steps  and  model  development 


1  Engineer  Circular  1 105-2-412,  “Assuring  Quality  of  Planning  Models,”  USACE  2011 

http://140.194.76.129/publications/eng-circulars/EC  1105-2-412  2011Mar/EC  1105-2- 

412  2011Mar.pdf 
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objectives,  and  contributes  to  the  credibility  of  work  products  (e.g.,  studies 
and  projects)  that  rely  on  information  yielded  by  the  model.  The  term 
“model  development  team,”  used  throughout  this  document,  represents  the 
group  of  individuals  involved  in  the  development  of  the  model.  Model 
development  teams  should  be  composed  of  modelers,  project  managers, 
subject  matter  experts,  engaged  stakeholders,  and  other  interested  parties. 
Model  development  should  be  coordinated  with  the  PCX  for  any  certifica¬ 
tion  requirements  as  laid  out  in  EC  1105-2-412  (USACE  2011).  As  explained 
below,  a  collaborative  and  iterative  approach  to  model  development  is 
essential  for  maintaining  transparency  and  scientific  defensibility. 


Plan  Formulation/Project  Objectives 


1 


Model  Application 


Figure  1.  Diagram  depicting  the  modeling  process.  Straight  arrows  indicate  flow  of 
process,  curved  arrows  represent  the  iterative  nature  of  modeling. 
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4  Conceptual  Modeling 

Conceptual  models  are  qualitative,  diagrammatic  summaries  that  describe 
important  components  of  a  system  and  the  interconnectedness  among  those 
components.  Conceptual  models  help  identify  how  drivers  and  stressors  will 
impact  system  dynamics  and  are  useful  blueprints  for  developing  quantita¬ 
tive  models  that  can  project  those  dynamics.  Conceptual  models  are 
incredibly  useful  because  they  force  stakeholders  to  visualize  the  system  in  a 
precise  way  (Fischenich  2008).  Given  the  complexity  of  environmental 
systems  and  the  sheer  abundance  of  factors  that  can  affect  natural 
processes,  these  models  provide  a  mechanism  to  organize  information  and 
also  serve  as  a  heuristic  tool  that  can  facilitate  discussions  across 
disciplinary  boundaries.  It  is  almost  impossible  to  develop  a  common/ 
collective  understanding  of  the  system  of  interest  and  develop  quantitative 
models  without  first  conceptualizing  the  system. 

This  section  focuses  on  describing  a  generalized,  good-practice  approach  for 
developing  conceptual  models  that  can  then  be  used  as  the  foundation  for 
developing  quantitative  models.  For  more  detailed  descriptions  of  the  use 
and  application  of  conceptual  models,  refer  to  Fischenich  (2008),  which 
describes  different  types  of  conceptual  models  and  identifies  general  good 
practice  guidance  for  their  use  in  restoration  project  planning.  When 
developing  conceptual  models  as  templates  for  quantitative  models,  six 
general  steps  should  be  followed  (described  in  detail  below): 

1.  Precisely  define  objectives  and  criteria  for  evaluation. 

2.  Bound  the  system  of  interest. 

3.  Represent  the  conceptual  model. 

4.  Describe  the  expected  patterns  of  model  behavior. 

5.  Identify  data  quality  and  quantity. 

6.  Identify  context  for  model  use. 

Precisely  define  objectives  and  criteria  for  evaluation 

The  initial  focus  of  conceptual  model  development  is  to  identify  and  review 
information  needs  and  expectations  of  those  engaged  in  a  particular 
(planning)  study,  and  formulate  objectives  to  guide  model  development 
efforts  and  manage  expectations.  The  objectives  must  be  defined  as 
precisely  as  possible  because  it  is  considerably  easier  to  develop  a 
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conceptual  model  (and  later  a  quantitative  model)  if  the  objectives  of  the 
model  application  have  been  delineated  clearly.  Solicitation  of  input  from 
local  subject-matter  experts  at  this  stage  can  be  beneficial  because  they  can 
facilitate  the  integration  of  expert  information  into  the  model  design  and 
development. 

The  criteria  that  the  model  must  meet  should  also  be  specified,  for 
example: 

•  The  model  should  have  a  structure  that  reasonably  portrays 
relationships  between  the  resource(s)  of  interest  and  system  elements 
that  cause  or  contribute  to  changes  in  their  condition  and  distribution. 

•  Model  results  (or  conclusions  based  on  the  model’s  application)  should 
correspond  well  with  both  real-system  behavior  and  data  from  the  real 
system. 

•  The  model  can,  once  fully  developed,  effectively  differentiate  between 
project  alternatives  that  differentially  affect  elements  that  cause  or 
contribute  to  changes  in  the  resource(s)  of  interest. 

•  Model  results  should  help  to  guide  metric  selection  to  gauge 
environmental  response  to  management  actions. 

It  is  important  to  formally  document  these  criteria  and  the  process/ 
participants  engaged  during  their  development  so  that  the  model 
development  team  and  others  can  refer  back  to  them  throughout  and 
following  model  development. 

Bound  the  system  of  interest 

Setting  an  appropriate  assessment  area  and  model  boundary  conditions 
can  be  challenging.  Environmental  systems  are  complex  and  certain 
components  may  not  be  important  for  a  particular  project,  but  only  a 
detailed  knowledge  of  the  system  can  identify  those  components. 
Furthermore,  every  model  can  be  conceptualized  in  myriad  ways  and  the 
goal  is  to  identify  those  components  that  provide  a  clear,  concise,  and 
informative  view  of  the  system,  given  the  objectives  of  the  project. 

Bounding  the  system  of  interest  consists  of  differentiating  between  those 
components  that  should  be  included  in  the  model  from  those  that  should  be 
excluded.  In  this  case,  components  refer  to  any  important  process  or 
variable  that  is  relevant  to  the  problem.  An  overly  complex  model  is  not 
desirable,  but  neither  is  excluding  components  that  might  be  critical  to  the 
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solution  of  the  problem.  Engaging  subject  matter  experts  is  crucial  to 
identifying  these  important  components.  Detailed  discussions  with  the 
experts  will  help  identify  what  should  be  included  in  the  conceptual  model, 
and  what  might  be  best  accounted  for  as  assumed  or  boundary  conditions 
(as  well  as  how  such  assumptions  might  affect/influence  later  model 
application).  Careful  thought  should  also  be  given  to  how  the  model  is 
bounded  in  time  and  space,  including  whether  a  regional,  rather  than  site- 
specific,  model  should  be  developed  (e.g.,  McKay  et  al.  (2011)).  If  the 
temporal  and  spatial  scales  are  too  broad,  then  too  much  attention  may  be 
focused  on  unnecessary  components  and  processes,  which  take  focus  away 
from  critical  system  processes.  The  documentation  for  this  step  should 
include  the  reasoning  and  logic  behind  choosing  which  model  components 
to  include. 

Represent  the  conceptual  model 

Formal  representation  of  conceptual  models  involves  creating  a  diagram  of 
the  system.  There  are  several  different  ways  to  represent  a  conceptual 
model,  ranging  from  simple  depictions  of  the  system  to  relatively  complex 
representations  written  in  a  modeling  language.  These  diagrams  play  an 
important  role  in  modeling  by  providing  a  “big-picture”  point  of  view  and 
more  importantly,  facilitating  communication  among  stakeholders.  This 
step  defines  how  those  components  and  variables  included  in  the  model  are 
interrelated.  One  of  the  most  common  types  of  conceptual  models  is  a  box- 
and-arrow  diagram,  which  provides  a  relatively  easy  framework  for 
representing  complex  environmental  systems:  variables  can  be  represented 
with  boxes  or  other  symbols  and  those  variables  that  interact  with  each 
other  can  be  connected  with  arrows.  Direction  and  strength  of  interactions 
and  levels  of  uncertainty  can  be  represented  as  well.  The  exact  symbology 
used  for  a  particular  modeling  effort  is  not  that  important;  however,  it  is 
paramount  to  maintain  precision  and  consistency  with  the  symbology  used 
within  a  given  model  (that  is,  make  sure  all  variables  of  a  certain  type  are 
drawn  the  same  way),  and  to  clearly  define  what  the  symbols  represent  (that 
is,  font,  line,  polygon,  and  image/symbol  styles  should  have  consistent 
meaning).  Many  different  techniques  are  used  to  draw  and  develop 
conceptual  models;  however,  a  detailed  description  of  these  techniques  is 
outside  the  scope  of  this  document.  Much  has  been  written  on  different 
approaches  (Starfield  and  Bleloch  1986,  Fries  et  al.  1998,  Ford  1999, 
Jorgensen  and  Bendoricchio  2001,  Ogden  et  al.  2005,  Fischenich  2008, 
Grant  and  Swannack  2008,  Jorgensen  and  Fath  2011).  Programs,  such  as 
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CEMCAT1  (Dalyander  and  Fischenich  2010)  have  been  developed  to 
facilitate  the  conceptual  modeling  process.  Documentation  at  this  stage 
should  include  not  only  the  conceptual  model,  but  also  a  detailed 
description  of  how  the  components  are  related,  including  rationale  and 
linkages  (i.e.,  explain  why  the  conceptual  model  was  constructed  the  way  it 
was)  (Killgore  et  al.  2008,  Casper  et  al.  2010).  At  this  point,  the  model 
development  team  should  refer  to  the  model  design  objectives  to  be  sure 
that  model  development  is  progressing  in  the  intended  direction  (and  if  not, 
document  the  rationale  for  deviations  from  those  objectives  and  engage  the 
eventual  users  of  the  model  to  inform  them  of  potential  impacts  on 
expectations). 

Describe  the  expected  patterns  of  model  behavior 

Subject-matter  experts  and  model  developers  will  always  have  some 
expectations  concerning  patterns  of  model  behavior.  These  expectations 
should  be  formally  described  and  thoroughly  documented  before  the  model 
has  been  quantified  for  two  main  reasons:  (1)  these  descriptions  can  serve 
as  points  of  reference  during  model  evaluation,  and  (2)  to  ensure  that  the 
expected  patterns  of  behavior  provide  the  types  of  projections  that  will  be 
useful  for  addressing  the  objectives  once  the  model  is  applied.  These 
expectations  can  be  based  on,  but  are  not  limited  to,  what  can  be  supported 
by  available  data.  Expert  opinion  (and  its  documentation)  is  critical  during 
this  step  because  subject  matter  experts  likely  know  more  about  relation¬ 
ships  among  model  variables  than  can  be  documented  in  a  rigorous  way  by 
data  alone.  Further,  for  environmental  systems,  there  are  always  important 
aspects  of  system  dynamics  for  which  there  are  no  data,  but  experts  may 
have  logical  and  valid  ideas  about  how  those  components  interact.  It  is 
common  to  formalize  these  expectations  as  graphs  representing  changes  in 
values  of  important  variables  over  time,  but  any  aspect  of  system  behavior 
for  which  there  is  an  expectation  should  be  noted  (for  example,  noting  the 
minimum  and  maximum  values  of  particular  variables,  proportional 
relationships  (as  variable  A  increases,  B  decreases),  etc).  Projected  patterns 
of  response  to  project  alternatives  should  also  be  documented  thoroughly. 
Consideration  should  be  given  to  the  cost-effectiveness  of  the  proposed 
scenarios  that  the  model  will  address.  Expectations  of  model  behavior  are 
frequently  left  implicit,  and  therefore  imprecise,  which  can  result  in  the 
model  not  being  designed  appropriately  and  therefore  being  incapable  of 
representing  important  aspects  of  system  behavior  that  are  needed  to 


1  http://cw-environment.usace.army.mil/eba/news.cfm?0ption=Title&Code=0&ld=821 
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address  the  objectives.  These  expectations  should  be  viewed  as  reference 
points  on  which  to  evaluate  the  model,  but  should  be  flexible  enough  to  be 
adapted  if  the  model  reveals  some  aspect  of  system  behavior  that  the  model 
development  team  had  not  thought  of. 

Identify  data  quality  and  quantity 

Once  the  conceptual  model  has  been  formulated  to  a  sufficient  degree,  it  is 
important  to  describe,  in  detail,  the  type  of  data  that  are  required  to 
parameterize  the  model,  then  to  determine  the  availability  of  those  data. 
Further,  any  uncertainties  associated  with  those  data  need  to  be  identified 
and  an  explicit  plan  must  be  formulated  on  how  to  deal  with  those 
uncertainties.  While  data  needs  are  often  known  before  the  conceptual 
model  has  been  developed,  the  conceptual  modeling  process  may  actually 
reveal  previously  unknown  data  requirements.  It  also  helps  to  focus  data 
acquisition  strategies  and  to  identify  critical  research  needs  that  can 
address  data  gaps,  which  may  also  provide  a  basis  for  targeting  data  needs 
for  monitoring  and  adaptive  management  planning  purposes.  Once  again, 
the  model  development  team  should  at  this  point  refer  to  the  model  design 
objectives  to  be  sure  that  model  development  is  progressing  as  intended. 
The  model  development  team  might  also  consider  reengaging  those  who 
intend  to  apply  the  model  to  discuss  any  changes  in  expectations,  and/or 
any  changes  that  might  be  warranted  based  on  information  learned  during 
model  development  activities  completed  to  date. 

Identify  context  for  model  use 

The  context  in  which  the  model  is  to  be  used  must  be  described  and 
documented  precisely,  which  includes  listing  all  of  the  restrictive  assump¬ 
tions  that  must  be  made  for  the  model  to  be  useful  for  projecting  system 
dynamics.  If  the  restrictive  assumptions  under  which  the  model  is  useful  are 
not  thoroughly  documented,  then  the  model  may  be  applied  inappropriately. 
It  is  important  to  note  that  an  exhaustive  list  of  assumptions  is  impossible, 
such  as  those  assumptions  that  can  be  left  implicit  (e.g.,  the  sun  will  continue 
to  rise).  Model  development  teams  should  focus  on  documenting  those 
assumptions  that  should  be  stated  explicitly  (e.g.,  minimum  flow  must  be  X) 
because  they  cannot  reasonably  be  assumed  to  remain  true/constant  under 
all  foreseeable  applications.  Documenting  the  appropriate  use  of  the  model  is 
a  critical  step  in  model  development  because  it  provides  the  framework  for 
model  development,  the  standards  for  model  formulation  and  evaluation,  the 
standards  for  model  application,  and  the  context  within  which  the  results  will 
be  interpreted  and  upon  which  its  applications  will  be  judged. 
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5  Quantitative  Model 

Given  that  several  types  of  models  are  used  for  projecting  environmental 
benefits,  as  discussed  above,  this  section  will  not  include  a  detailed 
description  of  using  particular  types  of  mathematics  or  statistics  for 
modeling  environmental  benefits.  Rather,  this  section  will  specify 
generalized  good  practice  for  developing  quantitative  models  of 
environmental  systems.  Quantitative  model  development  follows  five 
steps,  which  will  be  discussed  in  more  detail  below.  A  detailed  example  is 
provided  in  Chapter  7,  “Modeling  in  Practice  ” 

1.  Linking  to  the  conceptual  model. 

2.  Selecting  the  general  quantitative  structure,  time  unit,  and  spatial  scale  for 
the  model. 

3.  Identifying  functional  forms  of  model  equations. 

4.  Estimating  the  parameters  of  the  model  equations. 

5.  Executing  the  baseline  model. 

Linking  to  the  conceptual  model 

The  conceptual  model  discussed  above  should  be  used  as  the  template  for 
the  quantitative  model.  Too  often,  quantitative  models  do  not  have  a 
formalized  conceptual  component.  However,  tightly  linked  conceptual- 
quantitative  models  facilitate  not  only  model  development  and  evaluation, 
but  also  communication  among  stakeholders.  Each  linkage  between 
components  in  the  conceptual  model  should  be  represented  quantitatively. 
One  helpful  technique  is  to  label  the  conceptual  model  with  symbols  that 
refer  to  equations  used  in  the  quantitative  model. 

Selecting  a  general  quantitative  structure  for  the  model 

The  choice  of  a  particular  mathematical  style  depends  on  the  background 
and  experience  of  the  modeler,  the  intended  application(s)  of  the  model 
(e.g.,  what  will  the  model  results  be  used  to  demonstrate),  and  the  type  of 
model  being  developed.  For  example,  index-based  models  do  not  require 
complicated  mathematics,  whereas  large-scale  simulation  models  might 
use  more  involved  mathematics  such  as  a  series  of  differential  equations. 
Theoretically,  the  dynamics  of  the  system  should  be  able  to  be  reproduced 
equally  using  different  mathematical  techniques/methods.  Developers 
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should  pursue  a  general  strategy  of  developing  a  set  of  equations  that 
determine,  at  selected  points  in  time,  the  value  of  each  important  variable 
included  in  the  model. 

For  environmental  models,  it  is  critical  to  identify  a  time-step  between 
iterative  solutions  of  model  equations.  For  example,  if  tree  growth  is  an 
important  variable  of  interest,  then  forecasting  tree  growth  every  second 
may  not  be  useful,  whereas  forecasting  tree  growth  every  year  would 
probably  provide  a  more  appropriate  level  of  detail.  Time  units  need  not 
be  confined  to  familiar  units  such  as  l  day  or  l  year,  but  they  may  be 
defined  as  any  length  that  allows  the  model  to  address  the  objectives  and 
appropriately  represent  the  temporal  dynamics  of  the  system.  Choosing  a 
time  unit  is  not  trivial  -  if  the  time  unit  is  too  long,  the  model  may  not 
capture  some  of  the  important  processes,  whereas  if  it  is  too  short,  the  risk 
of  reducing  interpretability  is  increased  (e.g.,  there  is  little  point  in 
projecting  population  dynamics  on  a  daily  basis  if  the  dynamics  of  interest 
occur  on  an  annual  basis  and  when  the  important  question  is  population 
persistence  over  50  years).  Once  the  appropriate  time  unit  has  been 
chosen,  the  time  unit  and  logic  used  to  make  the  choice  need  to  be 
included  in  the  documentation1. 

Likewise,  it  is  important  to  identify  the  appropriate  spatial  scale  for  the 
model  output.  Environmental  processes  have  different  impacts  at  different 
spatial  scales,  so  it  is  important  to  consider  how  those  processes  should  be 
represented  quantitatively.  It  is  important  to  consider  not  only  direct 
impacts  of  a  project,  but  to  also  consider  indirect  impact.  For  example,  a 
stream  restoration  project  will  directly  impact  the  habitat  surrounding  the 
stream,  but  may  also  have  impacts  further  downstream.  Once  the 
appropriate  spatial  scale  is  chosen,  the  scale  and  logic  used  to  make  the 
choice  should  be  documented.  The  choice  of  a  spatial  scale  closely 
coincides  with  the  choice  of  the  temporal  scale.  Selecting  the  wrong  space 
or  time  scales  can  contribute  to  erroneously  low  or  high  values  of 
important  variables,  so  careful  consideration  must  be  given  to  both. 

Identifying  functional  forms  of  model  equations 

The  next  step  in  quantitative  model  development  is  to  determine  the 
functional  forms  of  the  model  equations— that  is,  determining  if  the  general 
forms  of  the  equations  representing  specific  relationships  are  linear, 


1  Note  that  the  typical  Corps  planning  horizon  is  50  years,  so  that  a  5-10  year  time-step  may  be 
sufficient  for  applications  in  feasibility  studies. 
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sigmoidal,  exponential,  etc.  Examples  are  illustrated  in  Chapter  7.  This  is 
analogous  to  choosing  which  type  of  statistical  analysis  to  perform  on  a 
dataset  (e.g.,  linear  regression,  non-parametric  tests,  etc.).  A  model  can 
contain  multiple  types  of  functional  forms,  and  the  more  complex  the 
model,  the  more  likely  that  it  will  contain  multiple  types  of  functional  forms. 

Four  general  types  of  information  can  be  used,  individually  or  in 
combination,  to  determine  a  functional  form  for  a  particular  relationship: 

(1)  quantitative  data,  (2)  information  based  on  theoretical  or  empirical 
relationships,  (3)  qualitative  information,  and  (4)  information  gained  from 
experimenting  with  the  model  itself.  Empirical  data  are  most  useful, 
particularly  if  available  from  within  the  system  of  interest.  However,  there 
are  almost  always  important  relationships  for  which  there  are  no  data  or 
observations.  In  these  cases,  the  scientific  literature  or  subject  matter 
experts  may  provide  sound  theoretical,  empirical,  or  qualitative  relation¬ 
ships  that  can  be  used  in  place  of  hard  data.  For  situations  where  data  and 
qualitative  understanding  are  lacking,  insight  can  be  gained  into  possible 
functional  forms  or  a  model  equation  by  hypothesizing  different  functional 
forms  and  observing  model  behavior  in  response  to  each.  Through  such 
experimentation,  the  choices  of  functional  forms  can  be  narrowed  by 
excluding  those  forms  that  produce  unreasonable  results.  Obviously,  the 
number  of  equations  that  can  be  identified  by  this  technique  (within  a  given 
model)  is  small  -  the  more  equations  specified  by  trial  and  error,  the  higher 
the  likelihood  that  reasonable  results  occur  only  by  chance.  It  is  important 
to  use  functional  relationships  that  are  interpretable  within  the  subject 
matter  of  the  project  (e.g.,  hydrologically,  environmentally,  etc).  If 
functional  relationships  are  not  documented  and  interpretable,  then  the 
ability  to  describe  overall  model  behavior  is  compromised.  As  a  matter  of 
practice  and  communication  with  non-modelers,  the  model  development 
teams  should  at  this  point  consider  revisiting  and  updating  conceptual 
models  to  reflect/document  the  manner  in  which  functional  forms  were 
developed. 

Determining  functional  relationships  can  be  difficult,  but  one  rule  of 
thumb  is  to  describe  the  functional  relationships  clearly  in  words  before 
describing  them  mathematically.  Modelers  and  stakeholders  should  take  a 
few  minutes  to  describe  the  model  verbally.  If  areas  of  the  model  are 
difficult  to  explain  verbally,  then  it  is  likely  that  those  areas  will  be  difficult 
to  explain  mathematically.  These  are  generally  the  areas  that  require  more 
thought  and  research.  Another  common  approach  for  dealing  with 
functional  relationships  is  to  describe  them  graphically.  Graphical 
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representations  provide  an  intermediate  step  between  verbal  and 
mathematical  representations.  If  functional  forms  are  unknown,  it  is  often 
useful  to  assume  linear  relationships  between  variables  as  a  first  step. 
Linear  functions  are  useful  for  ecological  models  when  trying  to  project 
trends  in  long-term  dynamics  and  when  the  general  relationship  between 
two  variables  is  understood  (e.g.,  variable  A  increases  when  variable  B 
decreases),  but  the  exact  form  is  not.  Once  more  information  is  gathered, 
these  functional  forms  can  be  changed.  Figure  2  provides  an  example  of 
how  to  describe  a  functional  relationship  verbally,  graphically,  and 
mathematically. 

Estimate  the  parameters  of  the  model  equations 

Information  on  which  to  base  parameterization  of  model  equations  comes 
from  the  same  four  sources  as  choosing  the  functional  forms.  In  fact, 
choosing  the  functional  forms  and  parameterizing  model  equations  are 
often  based  on  the  same  information.  However,  from  a  modeling 
standpoint,  these  should  be  viewed  as  distinct  events  because  the  former 
generally  has  more  profound  implications  concerning  environmental 
interpretations  of  model  structure  than  the  latter.  Stated  another  way,  in 
general,  the  functional  form  of  an  equation  controls  the  pattern  of  the 
results,  whereas  the  parameter  value  controls  the  magnitude. 

The  specific  methodologies  used  to  estimate  parameters  of  model  equations 
are  as  diverse  as  the  field  of  statistics  and  the  appropriate  methodology  will 
vary  depending  on  the  type  of  data  being  analyzed  (e.g.,  hydrologic  data  and 
population  data  require  two  different  approaches).  The  important  point  is 
simply  that  parameters  should  be  estimated  using  appropriate  statistical 
techniques.  Often  equations  resulting  from  these  statistical  analyses  actually 
become  a  part  of  the  model.  Statistical  or  data-driven  models  should  be 
applied  with  caution  when  the  conditions  being  modeled  are  outside  the 
dataset  from  which  the  model  was  derived.  For  example,  a  sediment  model 
for  a  particular  stream  that  was  based  on  discharges  ranging  from  1000  to 
5000  cubic  feet  per  second  (cfs)  might  not  be  reliable  when  assessing  flows 
in  excess  of  20,000  cfs.  In  the  shorebird  example  above,  the  relationship 
was  developed  based  on  habitat  availability  in  the  range  of  3,000  to 
6,000  ha  and  might  not  hold  when  one  is  much  outside  this  range.  By 
highlighting  and  tracking  these  sorts  of  factors/questions,  model  develop¬ 
ment  teams  can  help  others  prepare  the  model  for  review  and  appropriate 
application,  and  be  ready  to  offer  suggestions  for  model  improvements  that 
might  be  required  to  adapt  the  model  for  broader  application. 
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A  project  has  been  proposed  to  restore  salt  marsh  habitat  along  a  coastal 
region  to  create  a  more  natural  habitat  and  to  restore  spring  shorebird 
populations  along  the  coast.  Anecdotal  evidence  indicates  that  shorebird 
abundance  increases  if  cord  grass  ( Spartina  alterniflora )  is  not  allowed  to 
spread  into  the  mudflats1.  However,  the  team  is  not  quite  sure  how  to 
model  it.  They  understand  that  increasing  mudflat  habitat  should  increase 
bird  abundance.  The  verbal  or  written  model  in  this  case  would  be: 

For  every  hectare  of  habitat  we  restore  to  mudflats,  we  expect  an 
increase  in  shorebird  abundance 

The  team  realizes  that  this  verbal  model  does  not  contain  enough 
information  to  represent  the  relationship  mathematically,  but  it  can  be 
represented  graphically: 


After  the  team  understands  the  general  relationship,  they  consult  subject 
matter  experts  and  the  scientific  literature  and  learn  that  previous  salt 
marsh  restoration  projects  resulted  in  an  increase  of  approximately 
60  shorebirds  in  the  spring  with  every  hectare  of  habitat  restored.  The 
verbal  model  would  then  be: 

For  every  hectare  of  habitat  restored  (i.e.,  mudflat  habitat),  we  expect  an 
increase  of  A  —  nr26o  shorebirds  in  the  the  spring. 

The  graphical  model  would  then  be  reformatted  to  include  these  numbers. 

Figure  2.  Simplified  example  of  representing  model  components  verbally,  graphically,  and 

mathematically. 


1  This  example  is  based  on  data  presented  in  Stralberg  et  al.  (2004).  However,  the  numbers  used  in 
this  example  were  approximated  and  should  not  be  considered  factual. 
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700,000 


l/> 


Mudflat  Area  Available  (Hectares) 


In  this  case,  the  equation  of  the  line  from  the  reformatted  graphical 
relationship  represents  the  quantitative  model: 

Increase  in  spring  shorebird  abundance  =  60  *  ( hectares  of  habitat  restored ) 

This  equation  could  then  be  used  as  part  of  a  larger  model  or  incorporated 
into  a  benefits  algorithm  to  determine  the  environmental  benefits  of  the 
project.  This  example  illustrates  how  models  can  be  developed  using  verbal, 
graphical,  or  mathematical  descriptions.  Note  that  several  assumptions  are 
made  in  this  model,  two  of  which  are  (l)  there  is  an  unbounded  upper  limit, 
and  (2)  there  are  zero  birds  if  there  is  no  mud  flat.  It  is  important  to 
document  the  assumptions  so  that  the  model  development  is  transparent 
and  defensible. 

Figure  2.  (concluded). 

In  cases  where  no  quantitative  data  are  available,  qualitative  information 
from  the  literature  or  expert  opinion  can  be  used  to  establish  assumptions 
on  which  to  base  estimates  of  model  parameters.  This  may  seem  like  a 
much  less  rigorous  process  than  analyzing  data,  and  indeed  most  scientists 
and  engineers  feel  less  comfortable  quantifying  models  this  way.  However, 
relying  solely  on  quantitative  data  may  result  in  long  and  futile  searches 
for  data  that  do  not  exist.  Subject  matter  experts  generally  know  more 
about  a  system  than  can  be  confirmed  with  data.  The  ultimate  goal  is  to 
use  techniques  that  allow  the  model  development  team  to  appropriately 
interpret  the  available  information  to  quantify  important  relationships 
within  the  system  of  interest.  It  is  likely  that  more  mistakes  (in  model 
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projections)  will  be  made  by  excluding  important  system  processes  from 
the  model  because  of  a  lack  of  data  than  by  quantifying  qualitative  data  for 
missing  components. 

Execute  the  baseline  model 

The  baseline  model  represents  the  behavior  of  the  system  under  a  particular 
set  of  conditions  that  are  being  used  as  a  benchmark  or  standard  against 
which  to  compare  project  alternatives.  Note  that  the  model  is  still  in 
development  and  is  about  to  be  evaluated.  At  this  stage,  do  not  apply  the 
model  to  the  project  alternatives.  The  model  should  be  run  and  the  results 
used  during  the  model  evaluation  phase  (see  Chapter  6).  The  initial 
conditions  used  for  the  baseline  simulation  must  be  defined  carefully,  since 
these  conditions  are  often  used  as  a  point  of  reference  for  both  evaluating 
the  model  and  comparing  alternatives. 
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6  Model  Evaluation 

The  goal  of  model  evaluation  is  to  determine  if  the  model  is  acceptable  for 
its  intended  use,  i.e.,  that  it  is  useful  for  addressing  the  objectives  of  the 
project  (Rykiel  1996).  At  this  point,  the  model  should  not  be  considered 
complete  because  it  has  not  undergone  a  rigorous  evaluation  process.  Model 
evaluation  involves  evaluating  all  aspects  of  the  model,  including  theory, 
computer  code,  parameter  values,  model  behavior,  and  comparisons  with 
data  (Glaser  and  Bridges  2007,  Grant  and  Swannack  2008).  The  model 
development  team  should  be  intimately  involved  with  the  evaluation 
process.  The  model  evaluation  procedures,  the  model’s  performance  at  each 
stage  of  the  evaluation  process,  and  any  actions  taken  pursuant  to  evalua¬ 
tion  procedures  should  be  thoroughly  documented.  The  importance/ 
significance  of  good  documentation  practices  (i.e.,  organized  and  sufficient 
documentation  of  information  used  to  support  decisions  made  during 
development  and/or  to  demonstrate  reliability  of  model  results)  cannot  be 
overstated.  This  documentation  will  provide  model  users,  as  well  as  outside 
reviewers,  a  greater  understanding  of  how  and  why  the  model  behaves  the 
way  it  does.  Given  the  wide  range  of  potential  types  of  models  that  can  be 
used  for  modeling  environmental  benefits,  each  with  their  own  set  of 
appropriate  evaluation  criteria,  this  section  will  focus  on  describing  how  to 
thoroughly  examine  the  characteristics  of  a  model  that  make  it  useful. 

Given  the  wide  range  of  disciplines  that  use  models,  several  terms  are  used 
across  the  disciplines,  and  have  similar,  but  distinctly  different  meanings. 
For  example,  model  evaluation  has  also  been  termed  model  validation 
(Rykiel  1996)  or  corroboration  (Pascual  et  al.  2003).  For  example,  valida¬ 
ting  a  hydrodynamics  model  and  an  ecological  model  are  different.  Hydro- 
dynamic  models  must  meet  specific,  and  accepted,  criteria  if  the  model  is 
considered  “valid;”  however,  there  are  no  generally  accepted  standards  for 
validating  ecological  and  environmental  models  (Rykiel  1996,  Grimm  and 
Railsback  2005)  and  there  has  been  considerable  debate  within  the 
scientific  community  regarding  both  terminology  and  technique.  In  this 
manuscript,  model  evaluation  refers  to  the  rigorous  process  of  evaluating 
models  and  it  consists  of  several  steps,  one  of  which  is  validation.  Figure  3 
provides  a  more  detailed  description  of  terminology  used  in  evaluating 
models.  This  section  of  the  report  provides  generalized  good  practice 
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guidance  for  evaluating  models.  The  baseline  model  is  evaluated  following 
five  steps,  each  of  which  will  be  described  in  more  detail  below.1 


The  terms  calibration,  verification,  and  validation  have  different  meanings 
across  disciplines.  When  performing  a  model  evaluation,  it  is  important  that 
the  terminology  being  used  is  defined  precisely  by  the  model  development 
team.  For  the  purposes  of  this  document,  the  terms  calibration,  verification, 
and  validation  will  be  defined  as  below. 

Calibration:  The  process  of  adjusting  model  parameters,  within  physically 
defensible,  and  ecologically  reasonable,  ranges,  until  the  resulting  predic¬ 
tions  give  the  best  possible  fit  to  the  observed  data.  In  some  disciplines, 
calibration  is  also  referred  to  as  "parameter  estimation." 

Verification:  Examination  of  the  algorithms  and  numerical  technique  in 
the  model  to  ascertain  that  they  truly  represent  the  conceptual  model  and 
that  there  are  no  inherent  numerical  problems  with  obtaining  a  solution. 

In  some  disciplines,  verification  is  also  referred  to  as  “code  testing.” 

Validation:  The  process  of  confirming  a  model's  applicability,  usually 
conducted  by  applying  a  calibrated  model  to  a  set  of  data  separate  from 
that  used  in  the  calibration  process  to  demonstrate  the  accuracy  of 
predicted  results.  In  some  disciplines,  validation  is  also  referred  to  as 
“evaluation,  skill/fitness  testing,  or  post- auditing.” 

Figure  3.  Common  terms  used  in  model  evaluation. 

1.  Evaluate  correspondence  between  model  results  and  expected  patterns  of 
model  behavior  (e.g.,  are  model  computations  correct  )(model  verification) 

2.  Examine  correspondence  between  model  projections  and  data  from  real 
system  (e.g.,  does  the  model  adequately  emulate  characteristics  of  the  real 
world)(model  validation) 

3.  Adjust  empirical  parameters  or  model  coefficients  to  match  a  known 
behavior,  expert  opinion  or  reference  site  data  (e.g.,  modify  model 
parameters  so  that  it  adequately  emulate  characteristics  of  the  real  world) 
(model  calibration) 

4.  Determine  levels  of  uncertainty  associated  with  model  forecasts 

5 .  Identify  data  gaps  and  research  needs  that  may  not  have  been  obvious 
during  conceptual  model  development 


1  Note  that  different  discplines  evaluate  models  in  sequential  orders  different  than  presented  here.  The 
order  itself  (particularly  steps  1-3)  are  not  “set  in  stone."  Most  model  evaluation  is  done  iteratively 
and  the  steps  can  be  cycled  through  more  than  once. 
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Evaluate  correspondence  between  model  results  and  expected 
patterns  of  model  behavior  (model  verification) 

Model  behavior  is  always  a  result  of  the  rules  written  in  mathematics  or 
computer  code,  but,  surprisingly  model  behavior  can  result  from  several 
things:  conceptual,  logical,  or  computing  errors  or  the  interactions  among 
model  components.  Therefore,  model  behaviors  always  need  to  be 
explained  before  being  accepted.  This  step  involves  comparing  model 
results  to  the  a  priori  expectations  identified  during  the  fourth  step  of 
conceptual  model  development  (i.e.,  “describe  the  expected  patterns  of 
model  behavior”).  In  these  comparisons,  look  for  obvious  impossibilities, 
such  as  negative  values  for  variables  that  must  be  positive  or  implausibly 
high  or  low  values  for  particular  variables.  While  this  may  seem  trivial, 
models  that  show  gross  inconsistencies  with  the  expectations  are  common 
(Figure  4).  These  inconsistencies  may  result  from  fundamental  misconcep¬ 
tions  about  the  nature  of  the  relationships  within  the  system.  Or,  they  may 
result  from  erroneous  expectations,  which  were  illuminated  from  the  model 
itself.  In  either  case,  the  model  development  team  is  obligated  to  reconcile 
differences  either  through  re-conceptualizing/requantifying  the  model  or  by 
adapting  the  expectations  based  on  the  new  knowledge  gained  from  the 
model. 

Once  a  model  no  longer  exhibits  obviously  implausible  behavior,  the  general 
dynamics  of  components  should  be  examined  to  ensure  that  the  timing  of 
maximum  and  minimum  values,  the  relative  amplitude  and  periodicity  of 
fluctuations,  and  relationships  with  other  variables  are  reasonable. 
Inadequacies  detected  as  a  result  of  this  closer  examination  may  still  be 
caused  by  fundamental  misconceptions  about  the  nature  of  the  relationship 
within  the  model  (that  is,  the  model  could  be  conceptually  flawed),  but  at 
this  point,  it  is  likely  that  the  inadequacies  resulted  from  erroneous 
parameter  estimates  or  perhaps  by  inclusion  of  incorrect  functional  forms. 
Understandably,  these  inconsistencies  can  be  fixed  by  adjusting  parameter 
values  or  functional  forms;  however,  it  is  important  to  remember  that 
functional  forms  need  to  have  environmental/ecological  interpretations  and 
should  not  be  adjusted  just  to  match  expectations.  Refer  to  the  section  titled 
“Adjust  empirical  paramenters  or  model  coefficients  to  match  a  known 
behavior,  expert  opinion,  or  reference  site  data”  for  a  detailed  description  of 
adjusting  parameter  values.  The  model  development  team  should 
thoroughly  document  the  methods  they  used  to  address  this  step. 
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Figure  4.  Graphical  depiction  of  comparing  (A)  model  results,  to  (B)  expected  patterns  of  model 
behavior.  In  this  example,  the  expected  patterns  of  model  behavior  do  not  match  the  results 
generated  by  the  model.  The  functional  relationships  in  the  model  that  were  used  to  generate  this 

behavior  must  be  reevaluated. 

Examine  correspondence  between  model  projections  and  data  from 
real  system  (model  validation) 

The  manner  in  which  model  projections  are  compared  to  data  from  the  real 
system  depends  on  the  specific  objectives  of  the  project,  the  type  of  model 
being  used,  and  the  type  of  data  available.  During  this  step,  the  model  is 
tested  against  an  independent  dataset  to  observe  how  well  the  model  fits 
those  data  (Figure  5).  Data  from  the  real  system  that  are  used  in  model 
evaluation  must  be  independent  of  data  used  to  develop  the  model  or  the 
model  cannot  be  rejected.  The  idea  behind  this  exercise  is  that  if  the  model 
provides  a  good  representation  of  the  system,  then  it  should  be  able  to 
represent  other  systems  or  conditions  equally  well  (Glaser  and  Bridges 
2007).  It  is  also  important  to  note  that  this  exercise  will  only  confirm  the 
model  behavior  under  the  range  of  conditions  exhibited  in  the  independent 
dataset.  For  environmental  systems,  it  is  often  difficult  to  obtain  an 
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Figure  5.  Examples  of  criteria  used  for  model  evaluation.  (A)  Comparison  of 
model  projections  (solid  line)  to  real  data  (dotted  line)  where  the  model 
results  and  real  data  compare  relatively  well.  (B)  Comparison  of  model 
projections  (solid  line)  to  real  data  (dotted  line)  where  the  model  results  and 
real  data  do  not  compare  well  and  parameter  values  may  need  to  be 
adjusted  in  order  to  generate  acceptable  patterns. 
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independent  dataset  to  validate  the  model.  One  common  approach  is  to 
divide  the  dataset  and  parameterize  the  model  with  some  of  the  data  and 
then  validate  it  with  the  other. 

Validation  is  always  required  to  understand  model  reliability  and  to 
quantify  uncertainty  in  the  model  results.  It  attempts  to  determine  which 
sources  of  uncertainty  should  be  considered  when  developing  management 
strategies.  Validation  results  can  be  used  to  determine  which  model 
revisions  would  be  needed  to  reduce  the  uncertainty.  Jorgensen  and  Fath 
(2011)  identified  three  pertinent  questions  that  should  be  asked  during 
validation: 

1.  What  is  the  uncertainty  of  the  data  used  to  build  the  model?  The 
uncertainty  associated  with  the  data  should  be  thoroughly  documented.  If 
there  is  a  high  degree  of  uncertainty,  then  statements  justifying  the  use  of 
those  data  should  also  be  included. 

2.  Do  the  observations  represent  a  wide  range  of  system  dynamics?  If  not, 
then  additional  data  collection,  under  a  wider  range  of  conditions,  should 
be  considered. 

3.  Are  some  important  processes  or  components  missing  or  described 
wrongly  in  the  model?  Part  of  this  question  can  be  answered  by  evaluating 
the  correspondence  between  model  results  and  the  expected  patterns  of 
model  behavior,  as  described  earlier  in  this  chapter;  however,  it  is  likely 
that  the  process  of  validation  will  reveal  further  inadequacies  in  the  model 
and  these  can  be  addressed  by  reevaluating  those  components  in  the  same 
manner  as  described  earlier. 

The  model  development  team  should  thoroughly  document  the  methods 
used  to  address  this  step. 

Adjust  empirical  parameters  or  model  coefficients  to  match  a  known 
behavior,  expert  opinion  or  reference  site  data  ( model  calibration). 

The  goal  of  this  step  is  to  improve  parameter  estimation  by  “fine-tuning” 
parameter  values  in  the  model.  Parameters  in  environmental  science  are 
rarely  known  as  exact  values,  and  unlike  physical  systems,  these  parameters 
are  not  constant  but  change  across  time,  space,  and  situation.  All  ecological 
models  are  simplifications  of  the  natural  world.  The  most  important 
components  and  processes  may  be  included,  but  the  model  will  not  account 
for  every  detail  -  the  influence  of  some  unimportant  components  and 
processes  can  be  accounted  for  by  the  calibration.  This  may  give  slightly 
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different  parameter  values  from  the  real,  but  unknown,  values  in  nature,  but 
this  difference  may  partly  account  for  the  influence  from  omitted  details 
(refer  to  Figure  5  for  an  example  of  model  results  where  the  parameter 
values  might  need  to  be  adjusted  to  achieve  a  better  match  between  model 
results  and  real  system  behavior). 

It  is  difficult  to  determine  beforehand  which  parameters  will  be  adjusted 
for  calibration  because  the  choices  of  parameters  that  improve  model 
behavior  depend  on  the  specific  interactions  among  components  within 
the  model.  Choosing  parameters  for  calibration  should  be  based  on  the 
degree  of  uncertainty  associated  with  each  parameter.  Those  parameters 
associated  with  high  levels  of  uncertainty  should  be  considered  for 
calibration  first.  Calibration  should  not  be  carried  out  randomly  if  more 
than  two  or  three  parameters  have  been  selected  for  calibration.  For 
example,  if  10  parameters  need  to  be  calibrated  and  the  uncertainties 
justify  testing  10  values  of  each  parameter,  then  the  model  needs  to  be  run 
to10  times,  which  is  intractable  (Jorgensen  and  Fath  2011).  A  common 
approach  is  to  vary  each  parameter  using  its  minimum,  maximum,  and 
reported  values  to  determine  how  the  model  behaves  when  the  parameter 
is  changed.  Parameters  should  be  varied  one  at  a  time  until  the  behavior  of 
the  model  is  understood.  The  model  development  team  should  thoroughly 
document  the  methods  they  used  to  address  this  step. 

Determine  levels  of  uncertainty  associated  with  model  forecasts 

After  verification,  validation,  and  calibration  of  the  model  have  been 
completed,  uncertainty  in  model  outputs  (i.e.,  forecasts)  remains  and  must 
be  dealt  with  in  the  context  of  environmental  decision  making.  Most 
ecological  systems  are  so  complex  that  it  is  completely  impractical  to  gather 
data  on  all  important  aspects  of  a  system.  As  a  result,  most  environmental 
models  include  varying  levels  and  types  of  uncertainty,  ranging  from 
parametric  uncertainty  (e.g.,  a  particular  parameter  value  may  be  associated 
with  a  large  confidence  interval  and  the  true  value  of  that  parameter  is 
unknown)  to  structural  uncertainty  (e.g.,  having  to  hypothesize  how 
components  are  related).  Therefore,  techniques  have  been  developed  to 
evaluate  the  levels  of  uncertainty  present  within  a  model.  At  this  stage  in 
model  development,  most  of  the  structural  uncertainty  should  have  been 
dealt  with,  but  there  is  often  considerable  parametric  uncertainty  in  the 
system.  Given  the  large  range  of  uncertainty  associated  with  environmental 
parameters,  it  is  important  to  describe  in  detail  how  that  uncertainty 
propagates  through  the  model.  The  general  goal  is  to  determine  the  degree 
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of  response  of  model  behavior  to  changes  in  various  model  components.  By 
identifying  parameters,  relationships,  or  submodels  to  which  the  model 
behavior  is  the  most  responsive,  uncertainty  analysis  may  provide  an 
indication  of  the  relative  accuracy  with  which  each  parameter  or  relation¬ 
ship  should  ideally  be  estimated  (and  help  potential  users  to  identify  or 
differentiate  between  reducible  and  irreducible  forms  of  uncertainty 
embedded  in  the  model). 

Mathematical  models  developed  to  study  ecosystems  do  not  reveal  “the 
truth,”  they  merely  provide  an  approximation  of  the  phenomena  being 
modeled  based  on  incomplete  knowledge.  The  reasons  for  this  are  many, 
but  are  based  in  part  on  the  fact  that  understanding  of  ecosystem  processes 
is  limited,  and  ecosystems  themselves  are  decidedly  stochastic.  Thus, 
ecological  models  are  built  under  uncertainties  in  the  values  of  the  factors 
(e.g.  the  growth  rate  of  a  specific  population),  in  the  parameterization  of  the 
system  (e.g.  the  boundary  conditions  of  the  dynamics),  and  in  the  choice  of 
mutually  exclusive  scenarios  (e.g.  the  choice  of  equations  that  describe 
dynamics).  Identifying  and  characterizing  the  uncertainty  associated  with 
ecological  models  is  a  necessary  precursor  to  decision-making  based  on  the 
model  results  (Suedel  et  al.,  in  preparation).  One  technique  to  help 
understand  how  these  uncertainties  might  influence/affect  model  validation 
and  eventual  application  is  a  “sensitivity  analysis.” 

Sensitivity  analysis  aims  at  establishing  the  relative  importance  of  the 
input  factors  employed  in  a  model,  answering  questions  such  as: 

•  “Which  variables  have  the  greatest  effect  on  model  results?” 

•  “Which  of  the  uncertain  input  factors  are  more  influential  in 
determining  the  variability  affecting  the  inference?” 

•  “Which  input  could  be  eliminated  with  the  least  effect  on  the  variance 
of  the  output  of  interest?” 

•  “Are  there  factors  whose  effect  on  the  output  is  so  low  that  they  can  be 
confidently  fixed  anywhere  in  their  ranges  of  variation  without 
affecting  the  results?” 

McKay  and  Pruitt  (in  preparation)  show  how  sensitivity  analysis  can  be 
used  to  address  these  questions  for  a  case  study  involving  the  Wetlands 
Value  Assessment  model. 
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The  steps  of  a  sensitivity  analysis  are  similar  to  those  of  the  calibration 
process;  however,  they  are  significantly  different  in  interpretation. 
Calibration  refers  to  the  fine-tuning  of  parameters  to  determine  accurate 
values.  Sensitivity  analysis  seeks  to  disclose  how  perturbations  of  model 
elements  (inputs,  parameters,  or  even  algorithms)  affect  responses  in 
outputs  (and  which  elements  provoke  the  strongest  responses).  For 
example,  if  a  model  is  highly  sensitive  to  changes  in  a  particular  variable 
(that  is,  if  the  results  fluctuate  dramatically  depending  on  the  value  of  an 
important  parameter),  this  can  (but  does  not  always)  translate  into  a  lack 
of  confidence  in  model  projections,  particularly  if  the  available  input  data 
are  of  poor  quality  or  accuracy.  There  are  several  ways  to  deal  with  a  lack 
of  confidence  in  model  projections.  Multiple  parameterizations  of  the 
model  can  be  applied  to  the  specific  problem,  each  having  a  different 
estimate  for  the  given  parameter(s).  Another  option  is  to  represent  the 
uncertain  parameter  as  a  random  variable,  with  the  degree  of  variability 
reflecting  the  level  of  uncertainty  in  the  estimates,  and  then  run  several 
repetitions  of  the  model  to  generate  average  system  behavior. 

The  breadth  of  uncertainties  in  the  model  that  were  discovered  during  the 
uncertainty  analyses  must  be  thoroughly  documented,  so  that  model  users 
can  understand  the  conditions  in  which  the  model  behaves  reasonably. 
Model  projections  are  less  accurate  when  the  variables  are  parameterized 
using  conditions  that  exceed  the  dataset  from  which  the  model  was  derived. 

Identify  data  gaps  and  research  needs  that  may  not  have  been  obvious 
during  conceptual  model  development 

The  last  step  in  model  evaluation  is  to  identify  any  data  gaps  and  research 
needs  that  arose  during  model  evaluation.  One  common  occurrence  during 
model  evaluation  is  that  data  gaps  that  were  not  obvious  during  conceptual 
and  quantitative  model  development  reveal  themselves.  This  allows 
stakeholders  to  identify  future  research  needs  or  changes  to  the  present 
model  to  accommodate  data  availability.  Further,  the  process  of  rigorously 
evaluating  a  model  causes  the  model  development  team  to  think  about  the 
system  in  a  new  way,  which  may  result  in  reconceptualizing  or  re- 
quantifying  some,  or  all,  of  the  model.  If  that  is  the  case,  then  the  subject 
matter  experts  should  incorporate  this  new  knowledge  into  the  model 
framework  and  evaluate  it  using  the  same  processes  described  above. 
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7  Modeling  in  Practice 

Background 

In  theory,  model  development  proceeds  smoothly  from  conceptual  model 
development  through  model  application;  however,  this  rarely  occurs  in 
practice  and  quantifying  models  of  environmental  systems  can  seem 
overwhelming.  Modeling  is  an  iterative,  dynamic  process  (Odum  1984,  Ford 
1999,  Grant  and  Swannack  2008)  and  environmental  models  are  best 
developed  through  an  iterative  approach  where  a  preliminary  conceptual 
model  is  developed;  then  a  small  section  of  that  conceptual  model  is 
quantified  and  evaluated,  addressing  challenges  or  incorporating  new  ideas 
as  they  occur;  then  a  new  piece  of  the  conceptual  model  is  quantified  and 
evaluated;  and  so  on,  until  the  entire  conceptual  model  is  represented 
quantitatively.  Model  development  is  most  successful  when  the  model  is 
constructed  in  this  manner  collaboratively  with  modelers,  subject  matter 
experts,  and  the  eventual  end-users  of  the  model.  Unfortunately,  this 
iterative  approach  to  model  development  is  seldom  documented,  but  each 
of  the  practical  activities  can  be  directly  related  to  the  three  steps  mentioned 
above  (conceptualize,  quantify,  evaluate).  As  the  model  development  team 
quantifies  each  piece  of  the  conceptual  model,  they  are  forced  to  constantly 
reevaluate  the  model,  both  conceptually  and  quantitatively.  Often  the  pieces 
that  fit  well  together  conceptually  do  not  make  sense  after  performing  a 
quantitative  evaluation  of  those  pieces.  However,  by  quantifying  and 
evaluating  small  pieces  of  the  model  separately,  the  process  becomes 
significantly  easier  and  this  provides  greater  insight  into  system  dynamics 
and  greatly  reduces  the  likelihood  of  mathematical  or  logical  errors.  This 
chapter  provides  good  practice  guidance  on  how  to  proceed  through  the 
modeling  process.  An  example  is  provided  in  Figure  6  to  further  illustrate 
the  concepts. 

In  practice,  the  first  step  of  modeling  is  to  develop  a  preliminary  conceptual 
model  of  the  entire  system,  following  the  steps  discussed  in  conceptual 
model  formulation.  Once  the  conceptual  model  has  been  represented,  the 
next  step  is  to  outline  and  document  a  “plan-of-attack”  for  quantifying  the 
model.  Then,  identify  a  series  of  intermediate  developmental  models  (IDM) 
that  will  be  quantified  sequentially  until  the  final  model  has  been 
completely  quantified. 
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Once  the  plan  of  attack  has  been  outlined,  the  next  step  is  to  begin 
quantifying  and  evaluating  the  series  of  IDMs  (Grant  and  Swannack 
2008).  There  is  not  a  particular  order  in  which  IDMs  should  be  developed 
-  in  general,  the  first  IDM  should  be  trivially  simple  and  subsequent  IDMs 
should  be  increasingly  complex.  Making  each  addition  to  subsequent 
IDMs  as  simple  as  possible  facilitates  the  identification  and  correction  of 


ERDC/EL  TR-12-18 


33 


errors  and  also  promotes  understanding  of  the  relationships  within  the 
model.  Quantification  of  each  IDM  should  follow  the  steps  outlined  in 
Chapter  3,  “Model  Development.” 

As  each  IDM  is  built,  it  should  be  rigorously  evaluated  before  proceeding  to 
the  next,  following  the  steps  described  in  Chapter  6,  “Model  Evaluation.” If 
the  IDM  does  not  meet  the  evaluation  criteria,  then  it  should  return  to  an 
earlier  step  in  model  development,  either  quantitative  or  conceptual.  The 
most  common  form  of  adjustments  made  during  the  modeling  process  are 
discovering  that  the  conceptual  model  has  too  few  or  too  many  components 
or  that  the  functional  forms  of  model  equations  do  not  produce  reasonable 
results.  This  point  emphasizes  the  need  for  concurrent  documentation, 
evaluation,  and  review  during  model  development. 

Modeling  is  often  viewed  as  a  “black  box”  activity  and  there  may  be  several 
groups  of  people  that  do  not  trust  the  results  of  models.  The  model  develop¬ 
ment  team,  as  well  as  potential  users  of  the  model  or  its  information,  should 
be  engaged  as  early  as  possible  in  the  modeling  process  to  maintain 
transparency  and  foster  an  environment  in  which  trust  (between  developers 
and  users)  can  develop.  Thoroughly  documenting  each  IDM  is  incredibly 
important  because  as  the  IDMs  become  more  complex,  the  team  might 
want  to  revisit  an  earlier  version  and  proper  documentation  facilitates  this 
process.  Further,  documentation  provides  transparency  for  peer-reviewers, 
certification  reviewers,  policy  makers,  and  the  interested  public,  and  this  is 
critical  when  the  model  is  being  used  for  environmental  projects. 

Example  of  modeling  in  practice 

The  purpose  of  this  example  is  to  briefly  illustrate  how  to  both  develop  a 
model  using  the  IDM  approach  and  emphasize  the  iterative  nature  of  model 
development.  Assume  that  the  main  task  at  hand  is  to  develop  a  simple  model 
of  the  system  and  that  the  model  development  team  working  on  the  project 
has  identified  the  recruitment  of  seedlings  as  an  environmental  benefit  of  the 
restoration  project.  Note  that  the  model  presented  here  is  simple,  will  not  be 
fully  quantified,  and  is  only  intended  to  be  a  descriptive  illustration  of  the 
concepts  IDM  and  the  iterative  nature  of  modeling  in  practice. 

A  bottomland  hardwood  forest  lies  in  the  floodplain  of  a  dam- 
controlled  river.  Historically,  the  forest  was  inundated  in  the  spring 
and  essentially  dry  from  late  summer  through  winter.  During  high 
spring  flow,  the  forest  stored  water,  which  mitigated  flood  damage 
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downstream.  During  dry -down  events,  the  trees’ seeds  germinated  and 
new  recruitment  occurred.  Current  operations  have  prolonged  the 
annual  flooding  season,  preventing  new  seedlings  from  establishing 
themselves.  There  are  plans  to  alter  dam  operations  to  restore  the 
hydrology  and  habitat  to  more  natural  conditions.  One  of  the  proposed 
metrics  for  assessing  the  environmental  benefits  of  this  project  is 
increasing  seedling  recruitment,  which,  over  time,  would  create  a 
“healthier”  bottomland  hardwood  community.  The  objective  of  this 
project  is  to  develop  a  model  to  determine  if  changes  in  hydrology  lead 
to  an  increase  in  the  overall  number  of  seedlings  in  the  forest. 

Example:  Preliminary  Conceptual  Model 

The  first  step  for  solving  this  problem  would  be  to  develop  a  preliminary 
conceptual  model  that  represents  the  entire  system.  This  step  exactly 
follows  the  steps  described  in  Conceptual  Model  Development.  For  this 
example,  the  objective  has  been  identified,  so  the  next  step  is  to  bound  the 
system  of  interest.  There  are  myriad  considerations,  but  as  a  first  step,  the 
model  development  team  decided,  after  much  discussion,  that  the  system 
could  be  conceptualized  as  a  box-and-arrow  diagram1  having  two  major 
components:  the  depth  of  water  in  the  forest  during  the  growing  season  and 
the  number  of  seedlings  in  the  forest,2 3  represented  as  rectangles).  The  depth 
of  water  fluctuates  seasonally,  increasing  and  decreasing  as  water  enters  the 
floodplain  in  the  spring  and  leaves  in  the  summer  (indicated  by  arrows). 

The  team  decided  that  it  was  not  interested  in  where  the  water  came  from  or 
where  it  went  after  it  left  the  floodplain. 3  Likewise,  recruitment  is  a  seasonal 
process.  Quantitative  values  for  average  seedling  recruitment  and  historical 
seedling  mortality  are  available  and  these  variables  were  represented 
explicitly.  The  team  has  a  general  idea  about  how  much  water  depth 


1  In  general,  boxes  represent  points  of  accumulation  (or  storage  spaces)  and  can  be  thought  of  as 
storage  spaces  from  which  material  (in  this  case  water  or  biomass)  can  be  moved  in  and  out  (via  the 
darker  arrows  in  the  diagrams).  Lighter  arrows  indicate  directional  influence.  For  example,  the  water 
volume  on  the  floodplain  affects  tree  recruitment,  but  in  this  model,  tree  recruitment  does  not  affect 
water  volume. 

2  The  team  recognized  that  there  were  other  components  that  could  be  added  but  decided  that  other 
ecological  processes  were  not  directly  involved  with  seedling  recruitment.  Given  the  complexity  of  the 
system,  the  team  decided  that  the  first  step  should  be  to  create  a  simple  model  to  explore  the  overall 
dynamics  of  the  system,  then  develop  more  complex  models  if  needed. 

3  Sources  and  sinks  in  this  diagram  (represented  by  clouds)  represent  origination  and  termination 
points,  respectively.  These  points  represent  the  boundary  of  the  system  and  encompass  the  processes 
outside  the  bounds  of  the  system.  For  example,  the  team  decided  that  it  did  not  want  to  model  all  of 
the  intricacies  involved  with  tree  growth  (such  as  photosynthesis,  etc.),  rather  they  only  wanted  to 
consider  how  water  accumulation  affected  overall  biomass. 
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changes  in  the  floodplain  over  the  course  of  an  average  year  and  also  knows 
that  if  water  remains  on  the  floodplain  during  the  summer,  then  new  trees 
are  not  recruited  because  the  seeds  cannot  germinate.  However,  the  team  is 
not  sure  of  exactly  what  the  relationship  between  inundation  and  recruit¬ 
ment  looks  like,  so  they  decide  that  it  would  be  best  represented  as  an  index 
variable  (the  circle  labeled  recruitment  index).  Figure  7  depicts  the 
preliminary  conceptual  model. 


representing  the  hydrologic  component  of  the  system. 

(A)  represents  the  original  conceptualization,  and 
(B)  represents  the  reconceptualization  of  the  model  that 


includes  more  precise  information  regarding  the  relationship 
between  water  depth  on  the  floodplain  and  time  of  year. 

Once  the  conceptual  model  has  been  developed,  the  model  must  be 
documented.  At  this  stage,  several  assumptions  have  been  built  into  the 
model  (such  as  why  certain  variables  were  included  or  omitted  and  which 
variables  interact  with  each  other).  These  assumptions  must  be  recorded  in 
the  model  documentation.  As  mentioned  previously,  it  is  important  to 
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thoroughly  document  each  stage  of  model  development.  After  conceptual 
model  documentation  has  been  completed,  the  development  team  must 
describe  how  the  model  is  expected  to  behave,  identify  data  quality  and 
quantity,  and  describe  its  context  for  use  (Chapter  4). 

Example:  First  Intermediate  Developmental  Model  (IDMi) 

The  team  first  chose  to  quantify  the  relationship  between  Season  and  Water 
Depth  (Figure  2A).  The  team  decided  that  they  could  not  appropriately 
quantify  the  model  without  knowing  the  stage  of  the  river  at  particular 
times  and  based  on  this  knowledge,  they  realized  that  stage  must  be 
included  explicitly,  so  they  reconceptualized  their  system  to  include  river 
stage1  (Figure  2B).  The  team  realized  that  season  was  not  a  precise  time 
unit,  so  they  decided  that  week  of  year  would  work  best  for  both  the 
hydrodynamic  and  ecological  pieces  of  the  model  (Figure  2B).  The  team 
used  gauge  data  and  scientific  literature  to  develop  a  quantitative  hydro- 
dynamic  model  that  captured  the  processes  that  inundated  the  floodplain  at 
the  appropriate  depth  in  the  spring  and  was  dry  in  the  summer.  The  team 
then  evaluated  the  model  and  documented  the  equations,  following  the 
guidelines  in  Quantitative  Model  Development  and  Model  Evaluation. 

Example:  Second  Intermediate  Developmental  Model  (IDM2) 

Next,  the  team  developed  an  IDM  for  the  dynamics  of  the  trees  (Figure  3A). 
After  some  discussion,  the  team  decided  that  week  of  year  affected  both 
seedling  recruitment  and  normal  mortality.  Specifically,  the  team  knew  the 
approximate  time  of  year  when  the  seedlings  germinated  and  had  some  data 
available  that  indicated  seedling  mortality  changed  seasonally  (Figure  3B). 
The  team  then  quantified  this  IDM  by  using  scientific  literature  and 
available  data,  evaluating  and  documenting  each  step  as  they  went. 

There  are  two  important  items  worth  noting  for  this  IDM.  First,  seedlings 
are  only  recruited  during  a  specific  time  of  year,  so  the  model  must  include  a 
conditional  statement  that  only  allows  recruitment  during  the  appropriate 
season.2  The  second  is  that  the  team  knows  that  tree  recruitment  depends 


1  Reconceptualizing  a  model  is  a  common  occurrence  and  should  be  considered  a  normal  part  of  the 
modeling  process.  As  the  model  development  team  becomes  more  familiar  with  the  system,  they  will 
likely  realize  that  particular  components  of  the  model  might  be  better  represented  differently.  By 
following  the  IDM  and  iterative  approach  to  model  development,  reconceptualizing  the  system 
becomes  an  integral  part  of  the  modeling  process. 

2  Generally,  conditional  statements  are  written  as  IF-THEN-ELSE  statements,  where  the  model  checks  to 
see  if  a  condition  is  true  or  not.  In  this  case,  the  statement  could  be  written  as  IF  it  is  the  growing  season, 
TFIEN  seedlings  can  germinate,  ELSE  no  seedlings  (ELSE  can  also  be  interpreted  as  ‘otherwise’) 


ERDC/EL  TR-12-18 


37 


on  water  depth;  however,  this  IDM  does  not  contain  water  depth.  In  cases 
like  this,  it’s  best  to  use  a  surrogate  for  the  variable,  run  the  model  to  ensure 
that  it  is  working,  and  then  reparameterize  it  once  the  IDMs  are  linked  (see 
IDM3  for  a  better  description).  These  surrogate  variables  should  come  from 
the  scientific  literature.  For  example,  data  might  exist  from  other  systems 
that  could  be  used  as  a  surrogate  (for  example,  recruitment  from  a  similar, 
but  unaffected  floodplain).  Once  IDM2  was  completed,  it  generated  tree 
recruitment  and  death  seasonally,  as  was  reported  in  the  literature. 


ecological  component  of  the  system.  (A)  represents  the  original 
conceptualization,  (B)  represents  the  reconceptualization  of  the  model 
that  includes  more  precise  information  regarding  the  relationship 
between  tree  biomass,  recruitment,  and  time  of  year. 


ERDC/EL  TR-12-18 


38 


Example:  Third  Intermediate  Developmental  Model  (IDM3) 

The  final  IDM  represents  the  system  as  it  has  been  reconceptualized 
during  the  IDM  process  and  now  the  team  needs  to  link  IDMi  and  IDM2 
(Figure  9).  As  previously  mentioned,  the  team  understands  the  general 
relationship  between  tree  recruitment  and  water  depth  and  understands 
that  water  depth  affects  both  recruitment  and  mortality  (Figure  9).  The 
relationship  between  seedling  mortality  and  water  depth  has  been  well 
documented  for  this  particular  forest  system,  so  the  team  can  quantify  that 
relationship.  Although  quantitative  data  are  not  available  from  either  field 
work  or  the  scientific  literature  to  study  the  relationship  between  water 


complete,  reconceptualized  system,  after  two  intermediate 
developmental  models  were  completed. 
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depth  and  recruitment,  the  team  needs  to  determine  if  the  proposed 
project  will  enhance  seedling  recruitment.  One  common  technique  used  to 
solve  this  dilemma  is  to  create  an  index1  of  how  water  depth  affects  tree 
recruitment.  The  team  decided  that  the  number  of  weeks  the  floodplain 
was  inundated  would  be  a  good  metric  for  relating  water  depth  to  tree 
recruitment.  In  this  case,  an  index-based  approach  is  appropriate  because 
the  team  knows  the  general  quantitative  structure  of  the  relationship; 
specifically,  if  there  is  water  on  the  floodplain  during  the  growing  season, 
then  recruitment  decreases.  The  surrogate  value  for  tree  recruitment 
should  be  replaced  with  an  appropriate  value  before  the  model  is  run. 

There  is  some  uncertainty  regarding  exactly  how  recruitment  decreases  with 
increasing  inundation  (that  is,  the  functional  form  of  the  relationship  is 
unknown).  The  team  identified  three  potential  functional  relationships  that 
made  ecological  sense  for  the  variable  recruitment  index:  linear,  exponen¬ 
tial  decline,  and  sigmoid  (also  called  s-shaped  or  logistic)  (Figure  10). 
However,  the  team  could  not  decide  which  of  these  best  represented  the 
relationship  among  week  of  year,  water  depth,  and  tree  recruitment.  The 
most  appropriate  approach  is  to  thoroughly  evaluate  the  model  using  each 
of  the  three  relationships  (following  each  of  the  steps  in  Chapter  6,  “Model 
Evaluation ”).  Once  each  version  has  been  evaluated,  the  results  should  be 
compared  against  each  other  to  see  how  much  difference  there  is  among 
model  runs.  This  is  a  sensitivity  analysis  (Chapter  6)  because  the  model  is 
determining  sensitivity  to  changes  in  the  functional  form  of  a  particular 
variable.  There  are  four  general  outcomes  from  this  type  of  analysis: 

1.  None  of  the  model  runs  generated  results  that  make  sense. 

2.  One  or  more  of  the  runs  generated  completely  unreasonable  results,  while 
the  others  seemed  reasonable. 

3.  All  model  results  seem  reasonable,  but  the  results  are  significantly 
different. 

4.  All  model  results  seem  reasonable  and  are  very  similar. 


1 A  detailed  description  of  index-based  modeling  is  outside  the  scope  of  this  document.  Briefly,  the  index  is 
scaled  between  0  and  1,  where  1  represents  the  best  possible  conditions  and  0  represents  the  worst. 
This  scale  is  then  combined  with  other  variables  in  a  model,  using  whatever  mathematical  technique  the 
team  decides  is  best.  For  this  example,  1  would  represent  the  number  of  weeks  inundated  that  created 
the  best  situation  for  tree  recruitment  (represented  by  a  1).  The  shapes  of  the  lines  (the  functional  forms) 
represent  how  the  relationship  between  weeks  inundated  and  tree  recruitment  change  as  weeks 
inundated  changes.  Once  the  floodplain  is  inundated  for  a  certain  period  of  time,  no  trees  would  be 
recruited  (indicated  by  an  index  value  of  0). 


letween 
index  d< 
/ariable. 
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When  none  of  the  results  make  sense  (outcome  l),  then  the  functional  forms 
that  were  chosen  likely  do  not  represent  the  actual  relationship  in  nature,  or 
the  model  does  not  include  an  important  variable  or  process.  For  the 
former,  the  team  should  identify  other  functional  forms  to  evaluate.  If  the 
latter  is  the  case,  then  the  model  should  be  reconceptualized  to  include  the 
important  processes.  When  one  or  more  of  the  model  runs  seem  reasonable, 
but  others  do  not  (outcome  2),  then  the  model  development  team  should 
discard  the  functional  forms  that  did  not  generate  reasonable  results.  If  all 
of  the  model  results  seem  reasonable,  but  are  significantly  different 
(outcome  3),  then  the  model  development  team  must  decide  if  they  want  to 
move  forward  using  more  than  one  version  of  the  model  (where  each  ver¬ 
sion  is  a  model  containing  a  different  functional  form  for  the  index 
variable).  This  is  very  common  in  ecological  modeling  and  it  allows  the  team 
to  capture  some  of  the  uncertainty  associated  with  the  system  (applying 
multiple  versions  of  the  model  is  discussed  in  more  detail  below).  Finally,  if 
all  of  the  model  runs  generated  logical  and  reasonable  results,  which  are  not 
significantly  different  from  each  other  (outcome  4),  then  the  model  develop¬ 
ment  team  must  decide  if  using  just  one  functional  form  will  suffice  (that  is, 
they  can  choose  one  of  the  relationships  and  discard  the  others).  This 
process  must  be  documented  extensively  because  it  provides  a  record  of 
how  the  team  dealt  with  uncertainty  in  the  system.  Once  the  final  IDM  has 
been  thoroughly  evaluated  and  documented,  the  team  may  proceed  to 
Model  Application. 

Example:  Model  Application 

Once  all  of  the  IDMs  have  been  developed  and  evaluated,  the  model  is  ready 
to  be  applied  to  the  problem.  For  this  example,  the  proposed  project  will  be 
affecting  River  stage  and  the  model  will  be  forecasting  how  changes  in 
River  stage  affect  tree  recruitment.  The  team  will  need  to  develop  model 
runs  for  each  of  the  proposed  project  alternatives  and  compare  those  results 
to  a  future  without-project  scenario.  If  the  team  decided  to  move  forward 
with  multiple  versions  of  the  model  (see  outcome  3  in  IMD3),  then  the 
model  would  need  to  be  run  for  each  functional  form  (Figure  10)  for  each  of 
the  proposed  project  alternatives  and  also  for  the  future  without  project. 

The  team  then  compares  the  results,  documents  the  outcomes,  and 
determines  which  alternative  provides  the  most  benefits  (refer  to  Chapter  8, 
‘Model  Application”  and  Chapter  9,  “Model  Documentation ,  Quality 
Assurance,  and  Communication”  for  more  details). 
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Example:  Summary 

In  practice,  model  development  does  not  proceed  fluidly  from  the 
conceptual  model  to  model  application.  Rather,  models  are  developed  via 
a  series  of  intermediate  developmental  models,  each  of  which  has  a 
conceptual  diagram  that  is  used  as  a  template  for  quantification  and  a 
thorough  evaluation,  documenting  each  step  in  the  process.  This  process 
makes  it  easier  to  quantity  complex  models,  find  conceptual  or  logical 
errors,  and  develop  all  of  the  proper  documentation.  In  this  example,  the 
model  development  team  started  with  a  simple  conceptual  model  of  a 
system,  then  through  the  model  development  process,  realized  that  the 
simple  model  did  not  quite  capture  all  of  the  important  processes,  so  the 
team  reconceptualized  the  system  as  it  developed  the  model,  resulting  in  a 
more  precise  representation  of  the  system  (Figure  n).  The  resulting  model 
was  more  complex,  but  the  IDM  approach  facilitated  new  components 
being  integrated  into  the  model.  Recognizing  that  the  modeling  is  an 
iterative  process,  the  team  was  able  to  quickly  adapt  its  new  ideas  into 
model  development. 
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Figure  11.  Comparison  of  (A)  the  model,  as  originally  conceptualized  by  the  model  development  team,  and 
(B)  the  model  after  the  team  completed  the  modeling  process. 
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8  Model  Application 

The  goal  of  this  phase  of  model  development  is  to  apply  the  model  during 
execution  of  the  planning  study  or  project  activities.  For  environmental 
benefits  analysis,  this  most  often  entails  using  the  model  to  develop 
information  that  will  be  used  during  evaluation  and  comparison  of  various 
planning  alternatives.  During  an  investigation,  models  can  be  applied  in 
many  ways.  In  most  instances,  model  application  will  involve  three 
sequential  activities: 

1.  Define  project  alternatives. 

2.  Apply  model  to  alternatives  and  a  future-without-project  alternative  and  to 
any  alternative  scenarios. 

3.  Analyze  and  interpret  results. 

Definition  of  project  alternatives 

At  this  stage,  the  suite  of  study  baselines  and  preliminary  alternatives  have 
largely  been  defined  and  documented.  The  project  baselines  (future 
without-project  condition,  and  sometimes  the  existing  condition)  and  each 
of  the  alternatives  (or  at  least  an  initial  array  of  alternatives)  have  been 
developed  and  characterized  to  a  level  of  detail  sufficient  to  synthesize/ 
produce  data  or  other  information  necessary  to  run  the  model.  The  model  is 
applied  to  each  of  the  study  baselines  and  alternatives  by  those  familiar  with 
or  trained  in  the  application  of  the  model  in  accordance  with  its  prescribed 
standards  for  application.1  Each  baseline  and  alternative  typically  yields  a 
unique  set  of  data  (and  in  some  instances  other  parameters)  that  are  input 
into  and  processed  by  the  model  to  produce  outputs.  The  outputs  produced 
by  the  model  are  used  by  the  study/project  team  during  attempts  to 
characterize  conditions  that  one  might  expect  to  result  in  the  absence  (i.e., 
future  without-project  condition)  or  presence  of  (i.e.,  alternatives)  future 
federal  action.  The  conditions  of  each  alternative  are  typically  compared 
against  the  future  without-project  condition  to  identify  where  beneficial  and 
detrimental  impacts  might  be  occurring.  The  resulting  observations  may 
inform  decisions  to  proceed  with  a  specific  action  or  to  modify/ develop 


1  The  study  team  should  seek  to  fully  understand  assumptions  that  must  be  demonstrated  “true” 
throughout  the  model's  application.  Likewise,  the  study  team  should  seek  to  understand  and  account 
for  the  influence  of  boundary  conditions  (or  changes  in  boundary  conditions)  that  might  influence  the 
interpretation  or  credibility  of  observations  that  are  based  on  model  results. 
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alternatives  in  an  informed  manner  based  on  what  was  observed  during  the 
prior  suite  of  model  results.  During  this  iterative  process  of  alternative 
development/ refinement,  model  application,  and  results  evaluation/ 
comparison,  study/project  teams  should  use  all  the  tools  at  their  disposal  to 
understand  and  document  the  rationale  supporting  their  conclusions, 
recommendations,  and  decisions.  For  instance,  the  conceptual  model  (of  the 
model  to  be  applied)  could  be  used  by  the  study  team  to  anticipate  how 
measures  associated  with  a  proposed  alternative  might  affect  resources  of 
concern.  Other  best  practices  that  study  teams  might  consider  during 
development  and  documentation  of  model-informed  alternative  analyses 
include: 

•  Performance  and  consideration  of  results  of  uncertainty  analysis:  if  the 
model  has  a  high  degree  of  uncertainty,  it  is  likely  that  multiple 
iterations  of  each  alternative  should  be  run  to  encompass  the 
uncertainty  (refer  to  the  section  in  Chapter  7  titled  “Example  of 
modeling  in  practice”  for  example). 

•  Development  and  documentation  of  a  decision  context(s):  It  is 
important  to  have  a  clearly  defined  decision  context  in  mind  so  that 
sets  of  testable  model  alternatives  can  be  developed  in  a  logical,  rather 
than  shotgun  approach,  which  could  yield  misleading  (or  meaningless) 
information. 

•  Careful  design  of  model  runs  helps  to  mitigate  the  chance  that  the 
model  is  not  used  under  conditions  for  which  it  was  not  designed. 

•  Careful  and  organized  documentation  of  modeling  activities  for  later 
use  by  reviewers  and  decision-makers,  keeping  in  mind  that 
documentation  of  context  and  rationale  may  be  just  as  important  as 
documentation  of  actions  and  the  consequences  of  actions. 

Apply  model  to  all  alternatives 

Once  the  model  scenarios  have  been  identified,  apply  the  model  to  all  the 
alternatives,  including  a  future-without-project  alternative.  Results  from 
model  runs  should  be  saved  and  cataloged  so  that  they  are  easy  to  find, 
and  more  importantly,  easy  for  someone  else  to  use.  One  major  issue  with 
model  results  is  that  the  modeler  names  results  files  ambiguously,  making 
it  impossible  for  others  to  find  and/or  interpret  how  to  use  those  files. 
There  is  not  a  standard  convention  for  file  naming,  but  the  common 
approach  is  to  include  dates  and/or  times  in  the  filenames,  which  prevents 
files  from  being  overwritten.  The  filename  structure  should  be  included  in 


ERDC/EL  TR-12-18 


46 


the  documentation  so  future  model  users  understand  how  to  read  and  use 
model  output. 

Analyze  and  interpret  results 

Results  from  the  model  application  should  be  analyzed  using  several 
techniques  and  should  not  be  limited  to  a  statistical  test  determining 
significance.  First  results  among  model  runs  should  be  compared  to 
determine  if  there  are  any  significant  differences  among  model  runs. 
Subject  matter  experts  should  carefully  scrutinize  the  results  and 
determine  how  to  interpret  them  environmentally;  that  is,  are  the  results 
environmentally  significant?  Often  results  can  be  mathematically 
significant,  but  those  differences  detected  by  statistical  techniques  may 
not  affect  the  system  ecologically  (Glaser  and  Bridges  2007).  Ecological 
significance  will  vary  among  projects,  but  it  should  be  defined  and 
documented  for  each  situation.  Other  factors,  such  as  cost,  should  be 
considered  as  well.1 

Finally,  results  should  be  appropriately  interpreted  with  regards  to 
uncertainty  in  the  system.  If  the  model  has  proven  sensitive  to  changes  in 
parameter  values  (from  the  sensitivity  analysis),  then  the  results  should  be 
interpreted  in  terms  of  its  uncertainty.  That  is,  if  model  results  vary  over  a 
wide  range,  then  the  subject  matter  experts  should  determine  how  those 
results  affect  management  decisions  as  the  project  progresses. 


1  This  is  done  in  Step  4  of  the  planning  process  -  i.e.,  Evaluation  of  Alternatives  using  Cost  Effectiveness 
and  Incremental  Cost  Analysis. 
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9  Model  Documentation,  Quality  Assurance, 
and  Communication 

Throughout  the  modeling  process,  model  development  involves  clearly 
communicating  the  model  and  its  results  to  a  broad  group  of  interested 
stakeholders,  which  can  include  the  scientific  community,  the  appropriate 
Planning  Centers  of  Expertise,  other  agencies,  policy  makers,  potential 
users  of  the  model,  the  interested  public,  etc.  This  is  accomplished  through 
two  tasks:  (a)  documenting  the  model  by  describing  its  technical  aspects, 
and  (b)  explaining  how  to  use  and  interpret  the  model  and  its  results  to 
interested  parties.  Documenting  a  model  and  its  results  is  of  paramount 
importance  if  the  model  is  to  be  received  well  among  the  stakeholders. 
Documentation  should  be  created  during  model  development  and  not 
after  the  model  is  completed,  which  can  be  accomplished  with  interim 
reports  or  reviews. 

The  model  development  team  should  be  documenting  the  model  as  it  moves 
through  each  step  of  the  modeling  process  and  this  stage  involves  compiling 
the  documentation.  Model  documentation  requires  thoroughly  explaining 
the  conceptual  and  technical  aspects  of  how  the  model  was  built.  The 
problem  being  addressed,  the  specific  objectives,  the  information  base  being 
drawn  upon,  the  technical  method  used  to  analyze  the  information,  and  the 
results  and  conclusions  must  be  described.  There  must  be  an  under¬ 
standable  link  between  the  conceptual  model  and  the  quantitative  model. 
Data  sources  and  the  types  of  mathematics  used  should  be  explained.  More 
importantly,  the  assumptions  that  were  made  regarding  the  relationships 
among  variables  (i.e.,  functional  forms  and  parameter  values)  must  be 
explicitly  addressed.  Models  are  all  based  on  assumptions  and  this  informa¬ 
tion  must  be  thoroughly  documented  for  a  model  to  have  scientific 
defensibility.  The  limitations  of  the  model  should  be  explained  clearly  and 
concisely  and  the  appropriate  use  of  the  model  should  be  described  anytime 
the  model  is  requested  or  used.  As  mentioned  throughout  this  guide, 
documenting  the  model  as  it  is  built  facilitates  this  step.  The  ideal 
documentation  results  in  technically  competent  people  being  able  to 
recreate  the  model  from  its  description.  Model  documentation  should 
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follow  the  guidelines  established  for  model  certification  by  Headquarters, 
USACE.1 

Transmitting  the  model  to  interested  parties  has  always  been  a  difficult 
task.  Model  development  usually  requires  many  decisions  that  make  sense 
to  the  model  developers,  but  when  viewed  by  others  after  completion  of 
the  model,  may  not  make  much  sense.  Confidence  in  the  model  is  always 
higher  the  earlier  stakeholders  or  potential  users  become  involved  with  the 
process.  In  cases  where  interested  parties  cannot  be  involved  with  model 
development,  a  clearly  documented  model  is  helpful,  but  communication 
is  also  easier  if  emphasis  is  placed  on  the  environmental  interpretations 
and  general  trends  of  the  results  rather  than  the  specific  numerical  values. 
General  trends  usually  are  more  relevant  in  a  management  context  than 
numerical  output;  however,  modelers  have  a  tendency  to  become 
preoccupied  with  presenting  details  at  the  expense  of  a  clear  overview. 
Clear  and  concise  communication  of  a  model  is  crucial  if  it  is  to  be  used  for 
making  or  informing  policy. 


i  see  http://140.194.76.129/publicatiens/eng-circulars/EC  1105-2-412  2011Mar/EC  1105-2- 

412  2011Mar.pdf. 
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10  Summary  and  Additional  Information 

This  report  provides  good  practice  guidance  for  developing  quantitative 
environmental  models  to  model  environmental  benefits.  Modeling  efforts 
should  begin  by  precisely  indentifying  objectives  for  the  project.  Once  the 
objectives  have  been  identified,  a  conceptual  model  should  be  developed 
and  then  used  as  the  template  for  quantitative  model  development. 
Quantitative  modeling  is  a  dynamic  process  and  models  are  best  developed 
through  an  iterative  approach  where  a  preliminary  conceptual  model  is 
developed,  and  then  small  sections  of  that  conceptual  model  are  quantified 
and  evaluated  in  a  piecewise  fashion  until  the  entire  conceptual  model  is 
represented  quantitatively.  This  iterative  approach  facilitates  model 
development  and  documentation,  as  each  section  of  the  larger  model  is 
conceptualized,  quantified,  evaluated,  and  documented  as  it  is  built. 

Research  presented  in  this  technical  report  was  developed  under  the 
Environmental  Benefits  Assessment  (EBA)  Research  Program.  The 
USACE  Proponent  for  the  EBA  Program  is  Rennie  Sherman.  The  Technical 
Director  is  Dr.  Al  Cofrancesco,  and  the  Program  Manager  is  Mr.  Glenn 
Rhett  of  the  ERDC  Environmental  Laboratory.  The  authors  gratefully 
acknowledge  technical  reviews  and  suggestions  provided  by  Nathan 
Beane,  Carl  Cerco,  Sarah  Miller,  David  Price,  Bruce  Pruitt,  and  Antisa 
Webb  (ERDC),  Jeff  Tripe  (Kansas  City  District),  Shawn  Komlos  (IWR), 
John  Wright  and  Brook  Herman  (Chicago  District). 
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